Skip to contentSkip to conversation

Splitting kernel headers and deprecating __KERNEL__

RFC David Howells 49 participants 178 messages Nov 26, 2004 – Dec 16, 2004 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

What we've come up with is this:

(1) Create new directories in the linux sources to shadow existing include directories:

NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/user/ include/linux/
include/user-*/ include/asm-*/

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

(2) Take each file from the shadowed directory. If it has any userspace

     relevant stuff, then:

     (a) Transfer this stuff into a file of the same name in the new
	 directory. So, for example, the syscall number list from
	 include/asm-i386/unistd.h will be transferred to
	 include/user-i386/unistd.h.

     (b) Make kernel file #include the user file. So:

		[include/asm-i386/unistd.h]
		...
		#include <user-i386/unistd.h>
		...

     (c) Where a user header file requires something from another header file
	 (such as a type), that file should include a suitable user header file
	 directly:

		[include/user-i386/termio.h]
		...
		#include <user/types.h>
		...

     (d) stdint types should be used where possible.

		[include/user-i386/termios.h]
		struct winsize {
			uint16_t ws_row;
			uint16_t ws_col;
			uint16_t ws_xpixel;
			uint16_t ws_ypixel;
		};

     (e) These header files should be bounded with __USER_XXXXX_H conditionals:

		[include/user-i386/termios.h]
		#ifndef __USER_I386_TERMIOS_H
		#define __USER_I386_TERMIOS_H
		...
		#endif /*  __USER_I386_TERMIOS_H */

(3) Remove all #if(n)def __KERNEL__ clauses.

(4) Remove the -D__KERNEL__ from the master kernel Makefile.

(5) For userspace use (such as for glibc), the appropriate include/user*/

     directories should be selected and installed in /usr/include/ or wherever,
     and symlinks made. For example, on i386 arch boxes, you might find:

	SOURCE			INSTALLED AS
	======================	============
	include/user/		/usr/include/user/
	include/user-i386/	/usr/include/user-i386/
				/usr/include/linux -> user
				/usr/include/asm -> user-i386

(6) On multi-arch archs (such as ppc64 which can also support ppc), you might

     find:

	SOURCE			INSTALLED AS
	======================	============
	include/user/		/usr/include/user/
	include/user-ppc/	/usr/include/user-ppc/
	include/user-ppc64/	/usr/include/user-ppc64/
				/usr/include/linux -> user
				/usr/include/asm-ppc -> user-ppc
				/usr/include/asm-ppc64 -> user-ppc64

     And then /usr/include/asm/ might contain files that do arch-size dependent
     switching between user-ppc and user-ppc64.

David

#2 lore raw
In reply to Alexandre Oliva
Quoted 5 lines · Alexandre Oliva

This means these headers shouldn't reference each other as linux/user/something.h, but rather as linux/something.h, such that they still work when installed in /usr/include/linux. This may require headers include/linux/something.h to include linux/user/something.h, but that's already part of the proposal.

That's going to take severe brain-ache to get right ... and worse, keep right. These headers aren't going to get tested outside the kernel tree often. So we'll have missing includes and files that only work if the <linux/> they're including is a kernel one rather than a user one.

I'm not particularly stuck on the <user/> namespace. We could invent a better name. How about <kern/> and <arch/> to replace <linux/> and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards compatibility.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#20 lore raw
In reply to David Howells
Quoted 9 lines · David Howells

(2) Take each file from the shadowed directory. If it has any userspace

     relevant stuff, then:

     (b) Make kernel file #include the user file. So:

		[include/asm-i386/unistd.h]
		...
		#include <user-i386/unistd.h>
		...

We may also want a user-asm symlink pointing to user-$ARCH. If <linux/foo.h> wants a definition from <user-$ARCH/foo.h> then it has to include <asm/foo.h> which includes <user-$ARCH/foo.h>. If asm/foo.h is empty other than the include, then it'd be nice to delete it and have <linux/foo.h> include <user-asm/foo.h> directly.

Quoted 8 lines
     (c) Where a user header file requires something from another header file
	 (such as a type), that file should include a suitable user header file
	 directly:

		[include/user-i386/termio.h]
		...
		#include <user/types.h>
		...

It's occasionally been on my mind that the transition from linux -> asm should be one way and that asm files should not include linux files. I'm not sure this is necessarily a worthy goal, but it seems worth mentioning.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#3 lore raw
In reply to Matthew Wilcox
Quoted 4 lines · Matthew Wilcox

I'm not particularly stuck on the <user/> namespace. We could invent a better name. How about <kern/> and <arch/> to replace <linux/> and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards compatibility.

There's no _real_ need to keep them. All we need to fix is a handful of libc implementations; anything else using them was broken anyway.

Signature · 1 line
dwmw2
#21 lore raw
In reply to David Howells
Quoted 6 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

What we've come up with is this:

I've already done something vaguely along those lines, extracting user-visible bits from include/linux/mtd/*.h, putting them into files in include/mtd/*.h and including them from there (except for jffs2.h which currently does it backwards).

Signature · 1 line
dwmw2
#4 lore raw
In reply to David Woodhouse
Quoted 11 lines · David Woodhouse
Quoted 4 lines · Matthew Wilcox

I'm not particularly stuck on the <user/> namespace. We could invent a better name. How about <kern/> and <arch/> to replace <linux/> and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards compatibility.

There's no _real_ need to keep them. All we need to fix is a handful of libc implementations; anything else using them was broken anyway.

<rant mode>

Cute idea. So any new definition which is supposed to be used in userspace will take at least a year to propagate - well, I know, one can always look for the definition in the kernel headers, copy it, ... Now this is no solution.

And how do you want to deal with the fact that up to now all the netfilter headers required for userspace programming live in the kernel include tree? Now this has been like this for quite some years. Shall one no longer use netfilter?

It is easy to be happy with the status quo and to reject any attempt to change things for the better. If you really wanted to fix (g)libc you would have taken action long ago. You're around for long enough.

</rant mode>

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#5 lore raw
Quoted 4 lines · Andreas Steinmetz

And how do you want to deal with the fact that up to now all the netfilter headers required for userspace programming live in the kernel include tree? Now this has been like this for quite some years. Shall one no longer use netfilter?

Don't get me started on fucking netfilter. Have you tried running 32-bit userspace on a 64-bit kernel recently? Throw away the fucking netfilter shite and burn it.

Signature · 1 line
dwmw2
#6 lore raw
Quoted 9 lines · David Woodhouse
Quoted 1 line · Andreas Steinmetz

And how do you want to deal with the fact that up to now all the netfilter headers required for userspace programming live in the kernel include tree? Now this has been like this for quite some years. Shall one no longer use netfilter?

Don't get me started on fucking netfilter. Have you tried running 32-bit userspace on a 64-bit kernel recently? Throw away the fucking netfilter shite and burn it.

I know very well about this problem. Still, you can easily use 64-bit applications with regards to netfilter. If you are so annoyed about this problem I would suggest you contact the netfilter developers to find a solution :-)

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#7 lore raw
Quoted 4 lines · Andreas Steinmetz

I know very well about this problem. Still, you can easily use 64-bit applications with regards to netfilter. If you are so annoyed about this problem I would suggest you contact the netfilter developers to find a solution :-)

It's a shame _they_ didn't know well about this problem five or so years ago when the rest of us did. Oh well, it's on the list of things to attempt when I can actually stomach it.

In the meantime, don't even think about claiming that netfilter is an example of _anything_ which you can hold up as an example when we're talking about cleaning stuff up. Breaking it in its current state would be a feature, not a bug.

Signature · 1 line
dwmw2
#22 lore raw
In reply to David Howells
Quoted 1 line · David Howells

(b) Make kernel file #include the user file.

Does kernel really need to include user headers? When it is definition of some const then it should be defined in one file (to be sure it has only one definition). But user headers may have some compatibility hacks that kernel do not need (and even maybe does not want) to have.

How you will handle that?

Thanks,

Grzegorz Kulewski

#23 lore raw
Quoted 8 lines · Grzegorz Kulewski
Quoted 1 line · David Howells

(b) Make kernel file #include the user file.

Does kernel really need to include user headers? When it is definition of some const then it should be defined in one file (to be sure it has only one definition).

You have do define a interface between the kernel and the userspace..
you either include kernel headers from userspace (with a lot of __KERNEL__ in them) or you make separate headers with the definitions and include them in both kernel and userspace (better). BTW, these are not userspace headers like the ones in /usr/include, those are just special headers preparated so that they can be included both from the kernel and userspace.

Quoted 2 lines

But user headers may have some compatibility hacks that kernel do not need (and even maybe does not want) to have.

About the compatibility hacks.. now it's time to remove them, together with this change. I don't think this will happen before 2.7/2.8 and until then all should have changed their code.
If you announce these changes soon enough and the developers have enough time to change their code, I don't see any problems.
Maybe you also could wrap these definitions in some #ifdef's and mark them as deprecated and write somewhere that they'll be removed in the next stable tree (2.8). So you could check if a library compiles with the new headers or if it still uses some old definitions.

tom

#24 lore raw
Quoted 6 lines · Tomas Carnecky
Quoted 5 lines · Grzegorz Kulewski
Quoted 1 line · David Howells

(b) Make kernel file #include the user file.

Quoted 7 lines
Quoted 3 lines

Does kernel really need to include user headers? When it is definition of some const then it should be defined in one file (to be sure it has only one definition).

You have do define a interface between the kernel and the userspace..
you either include kernel headers from userspace (with a lot of __KERNEL__ in them) or you make separate headers with the definitions and include them in both kernel and userspace (better). BTW, these are not userspace headers like the ones in /usr/include, those are just special headers preparated so that they can be included both from the kernel and userspace.

Ok, so maybe do it in this way:
1. common headers (included by 2. and 3.)
2. kernel headers (things only for kernel + included 1.)
3. userspace headers (things only for userspace + included 1.)

This way we will have no ifdefs, one definition per thing, user code in userspace and kernel code in kernelspace only.

Quoted 6 lines
Quoted 2 lines

But user headers may have some compatibility hacks that kernel do not need (and even maybe does not want) to have.

About the compatibility hacks.. now it's time to remove them, together with this change. I don't think this will happen before 2.7/2.8 and until then all should have changed their code.
If you announce these changes soon enough and the developers have enough time to change their code, I don't see any problems.
Maybe you also could wrap these definitions in some #ifdef's and mark them as deprecated and write somewhere that they'll be removed in the next stable tree (2.8). So you could check if a library compiles with the new headers or if it still uses some old definitions.

Are you talking about breaking userspace (API and ABI) compatibility? And possibly breaking compatibility with older versions of standards? I do not think it could happen. (Well at least not for common widely-used APIs).

Instead we can place such userspace only hacks in 3.

Thanks,

Grzegorz Kulewski

#12 lore raw
In reply to Matthew Wilcox
Quoted 10 lines · Matthew Wilcox
Quoted 6 lines · David Woodhouse
Quoted 2 lines · David Howells

How about calling the interface headers "kapi*/" instead of "user*/". In case you haven't guessed, "kapi" would be short for "kernel-api".

I don't think that change really makes any difference. The nomenclature really isn't _that_ important.

Indeed. We could also make this transparent to userspace by using a script to copy the user-* headers to /usr/include. Something like this:

Indeed.

Quoted 3 lines

If we really wanted to get fancy, we could also sed __u32 to uint32_t. But that would probably cause more pain, confusion, hurt and bad feeling than I'd ever want to be responsible for.

Also true. Let's just use the standard types in the first place and not screw around with having to fix it up later.

Signature · 1 line
dwmw2
#27 lore raw
In reply to David Howells
Quoted 4 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

I am fully in support of both this idea and the current implementation you have described below.

Quoted 5 lines

What we've come up with is this:

(1) Create new directories in the linux sources to shadow existing include directories:

Quoted 7 lines

NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/user/ include/linux/
include/user-*/ include/asm-*/

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

I think they may be. If they are, they can be accommodated. For example:

include/user-scsi/ include/scsi/
include/user-net/ include/net/
include/user-rxrpc/ include/rxrpc/

etcetera. This only creates a conflict if someone creates a directory foo that also clashes with asm-foo. And if someone does propose that, we just kill them. Easy.

Quoted 9 lines
     (d) stdint types should be used where possible.

		[include/user-i386/termios.h]
		struct winsize {
			uint16_t ws_row;
			uint16_t ws_col;
			uint16_t ws_xpixel;
			uint16_t ws_ypixel;
		};

I really hate stdint. Can't we use __u16 instead?

Quoted 13 lines
     (e) These header files should be bounded with __USER_XXXXX_H conditionals:

		[include/user-i386/termios.h]
		#ifndef __USER_I386_TERMIOS_H
		#define __USER_I386_TERMIOS_H
		...
		#endif /*  __USER_I386_TERMIOS_H */

(3) Remove all #if(n)def __KERNEL__ clauses.

(4) Remove the -D__KERNEL__ from the master kernel Makefile.

(5) For userspace use (such as for glibc), the appropriate include/user*/

Quoted 9 lines
     directories should be selected and installed in /usr/include/ or wherever,
     and symlinks made. For example, on i386 arch boxes, you might find:

	SOURCE			INSTALLED AS
	======================	============
	include/user/		/usr/include/user/
	include/user-i386/	/usr/include/user-i386/
				/usr/include/linux -> user
				/usr/include/asm -> user-i386

This proposal doesn't address the asm-generic problem directly. Can I presume that you intend to also create linux/include/user-generic, install it as /usr/include/user-generic and create an asm-generic symlink that points to user-generic? A good problem file to be dealt with would be asm/errno.h

Quoted 14 lines

(6) On multi-arch archs (such as ppc64 which can also support ppc), you might

     find:

	SOURCE			INSTALLED AS
	======================	============
	include/user/		/usr/include/user/
	include/user-ppc/	/usr/include/user-ppc/
	include/user-ppc64/	/usr/include/user-ppc64/
				/usr/include/linux -> user
				/usr/include/asm-ppc -> user-ppc
				/usr/include/asm-ppc64 -> user-ppc64

     And then /usr/include/asm/ might contain files that do arch-size dependent
     switching between user-ppc and user-ppc64.

ppc might want to consider following the lead of parisc, s390 and mips and unify at least their header files, if not their arch directory.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#11 lore raw
In reply to David Woodhouse
Quoted 6 lines · David Woodhouse
Quoted 2 lines · David Howells

How about calling the interface headers "kapi*/" instead of "user*/". In case you haven't guessed, "kapi" would be short for "kernel-api".

I don't think that change really makes any difference. The nomenclature really isn't _that_ important.

Indeed. We could also make this transparent to userspace by using a script to copy the user-* headers to /usr/include. Something like this:

install_user_headers:

	$(srctree)/scripts/install_user_headers.sh

#!/bin/bash
rm -rf /usr/include/asm /usr/include/linux /usr/include/asm-generic mkdir /usr/include/asm /usr/include/linux /usr/include/asm-generic

to_user() {
	sed	-e 's,<user/,<linux/, \
		-e 's,<user-generic/,<asm-generic,' \
		-e 's,<user-[^/]*/,<asm,' \
		<$1 >$2
}

for src in include/user/*; do
to_user $src /usr/include/linux/basename $src
done

for src in include/user-$ARCH/*; do
to_user $src /usr/include/asm/basename $src
done

for src in include/user-generic/*; do
to_user $src /usr/include/asm-generic/basename $src
done

If we really wanted to get fancy, we could also sed __u32 to uint32_t. But that would probably cause more pain, confusion, hurt and bad feeling than I'd ever want to be responsible for.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#28 lore raw
In reply to Matthew Wilcox
Quoted 5 lines · Matthew Wilcox
Quoted 2 lines

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

I think they may be. If they are, they can be accommodated. For example:

Quoted 7 lines

include/user-scsi/ include/scsi/
include/user-net/ include/net/
include/user-rxrpc/ include/rxrpc/

etcetera. This only creates a conflict if someone creates a directory foo that also clashes with asm-foo. And if someone does propose that, we just kill them. Easy.

Agreed.

Quoted 11 lines
Quoted 9 lines
     (d) stdint types should be used where possible.

		[include/user-i386/termios.h]
		struct winsize {
			uint16_t ws_row;
			uint16_t ws_col;
			uint16_t ws_xpixel;
			uint16_t ws_ypixel;
		};

I really hate stdint. Can't we use __u16 instead?

We're trying to clean all this crap up. I think we'd need a better justification than 'I hate stdint' to use anything other than the standard types which the language provides.

You'll get over it, I promise. I hated stdint too for a whole week or so after I admitted it made sense to switch JFFS2 to it, because I actually wanted it to be usable other than in the Linux kernel.

Quoted 5 lines

This proposal doesn't address the asm-generic problem directly. Can I presume that you intend to also create linux/include/user-generic, install it as /usr/include/user-generic and create an asm-generic symlink that points to user-generic? A good problem file to be dealt with would be asm/errno.h

That makes sense.

Quoted 2 lines

ppc might want to consider following the lead of parisc, s390 and mips and unify at least their header files, if not their arch directory.

That makes a certain amount of sense, although ppc64 currently lacks the mess of platform-specific stuff that ppc32 has for various embedded platforms. I suspect it would take a fair amount of work to produce a single directory which everyone's happy with.

Signature · 1 line
dwmw2
#25 lore raw
In reply to Grzegorz Kulewski
Quoted 4 lines · Grzegorz Kulewski

Ok, so maybe do it in this way:
1. common headers (included by 2. and 3.)
2. kernel headers (things only for kernel + included 1.)
3. userspace headers (things only for userspace + included 1.)

Are there really any definitions that belong _only_ to userspace? AFAIK all that is in the kernel sources is used by the kernel, so there are no headers that would fit into your category 3. There is no reason to put such things into the kernel sources.

Quoted 3 lines

Are you talking about breaking userspace (API and ABI) compatibility? And possibly breaking compatibility with older versions of standards? I do not think it could happen. (Well at least not for common widely-used APIs).

Instead we can place such userspace only hacks in 3.

The API can be changed, I don't see any problems with the API
compatibility, if you document all changes then it's no problem for
the developers to change their code.
And the ABI? Well, I compile my kernel by myself, as I do all my userspace (gentoo). And the distributions still could define __DEPRECATED__ (or something similar).

I mean.. come on.. how hard is it to change some lines of code, some structure names, some function arguments.
I know that there's a problem if you have binary only files, but I doubt that anyone is running a exe that was compiled for against kernel 2.0 and every big company compiles their projects against each stable kernel line.
Oh.. and I forgot, you usually don't include the kernel headers in your 'hello world' program, most of the applications use (g)libc, one big library, so you'd need to change only very few projects, although I admit that can be rather big projects, but still...

BTW, how much smaller would the kernel be if we removed all the backwards compatibility? I mean the size of the compiled kernel image.

tom

#148 lore raw
In reply to David Howells
Quoted 4 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

Yay!!! Finally!!!

Quoted 4 lines
	NEW DIRECTORY		DIRECTORY SHADOWED
	=============		==================
	include/user/		include/linux/
	include/user-*/		include/asm-*/

How about include/abi for the platform-independent headers and include/arch-* for the platform-specific headers? Then include/arch would be symlinked to the correct arch-* directory.

Quoted 2 lines

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

Perhaps such directories should either be broken out into abi/foo<->linux/foo pieces, or put into their own top-level directories.

Quoted 3 lines

(2) Take each file from the shadowed directory. If it has any
userspace
relevant stuff, then:

[...snip...]

Something like this would warn about incorrect uses of the linux/ headers, yet still allow for a period of backwards compatibility. Once such period has passed, then linux/* would not even be installed, only abi/* and arch/*

include/linux/foo.h:
#ifndef _LINUX_FOO_H
#define _LINUX_FOO_H 1

#ifndef __KERNEL__

# warning "Directly including <kernel/foo.h> is deprecated!!!"
# warning "Please change your program to include <abi/foo.h> instead!!!"
#endif

#include <abi/foo.h>

#ifdef __KERNEL__
/* Old __KERNEL__ contents of include/linux/foo.h */ #endif

#endif /* not _LINUX_FOO_H */

include/abi/foo.h:
#ifndef _ABI_FOO_H
#define_ABI_FOO_H 1

#include <abi/bar.h>
#include <arch/baz.h>

/* Old non-__KERNEL__ contents of include/linux/foo.h */

#endif /* not _ABI_FOO_H */

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a17 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

#29 lore raw
In reply to David Woodhouse
Quoted 11 lines · David Woodhouse
Quoted 10 lines · Matthew Wilcox
Quoted 9 lines
     (d) stdint types should be used where possible.

		[include/user-i386/termios.h]
		struct winsize {
			uint16_t ws_row;
			uint16_t ws_col;
			uint16_t ws_xpixel;
			uint16_t ws_ypixel;
		};
Quoted 5 lines
Quoted 1 line

I really hate stdint. Can't we use __u16 instead?

We're trying to clean all this crap up. I think we'd need a better justification than 'I hate stdint' to use anything other than the standard types which the language provides.

The justification is that it doesn't properly describe the variable size (think userspace 32 bit variable vs. kernelspace 32 bit variable.) We need to stick with the proper __u32 type variables for data that crosses the userspace/kernelspace boundry.

See Linus's posts about this topic on lkml in the past for more details about this.

thanks,

greg k-h

#149 lore raw
In reply to David Howells

Salut,

Quoted 4 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

Fnord alert: you're trying to change the API and the ABI of Linux. The thing you fail to see is that Linux is not the only operating system in the world. Big companies are very prone of using their compatibility code to make their program run in any odd way they like without actually having to change anything. Also, you're talking about assumptions that have been true for decades.

Big companies are already eyeing Linux with fear because our kernels don't actually work and because we break a lot of compatibility in the stable series. If you now come up with something that breaks basically everything so they need a new libc and everything, it's rather likely that they'll run away.

I just want you to consider that. Linux needs to get to a stable state of affairs at some time.

Tonnerre

#8 lore raw
In reply to David Woodhouse
Quoted 2 lines · David Woodhouse

There's no _real_ need to keep them. All we need to fix is a handful of libc implementations; anything else using them was broken anyway.

It's not as simple as *fixing* them. They'd have to go through a transition period in which they support both formats, which probably implies a mess of ifdefs or, worse, macro expansion in #include directives.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#9 lore raw
In reply to Matthew Wilcox
Quoted 5 lines

I'm not particularly stuck on the <user/> namespace. We could invent a better name. How about <kern/> and <arch/> to replace <linux/> and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards compatibility.

I'd rather not change linux and asm within the kernel itself. We'd have to patch pretty much every file and move all the include files around. That's a lot of patch.

How about calling the interface headers "kapi*/" instead of "user*/". In case you haven't guessed, "kapi" would be short for "kernel-api".

Then we'd have in the kernel:

	NEW DIRECTORY		DIRECTORY SHADOWED
	=============		==================
	include/kapi/		include/linux/
	include/kapi-*/		include/asm-*/
	include/kapi-rxrpc/	include/rxrpc/
	include/kapi-net/	include/net/

And in userspace:

	SOURCE			INSTALLED AS
	======================	============
	include/kapi/		/usr/include/kapi/
	include/kapi-i386/	/usr/include/kapi-i386/
	include/kapi-generic/	/usr/include/kapi-generic/
				/usr/include/linux -> kapi
				/usr/include/asm -> kapi-i386
				/usr/include/asm-generic -> kapi-generic (?)

David

#1 lore raw
In reply to David Howells
Quoted 6 lines · David Howells
	SOURCE			INSTALLED AS
	======================	============
	include/user/		/usr/include/user/
	include/user-i386/	/usr/include/user-i386/
				/usr/include/linux -> user
				/usr/include/asm -> user-i386

Although user/ and user-* make a lot of sense within the kernel source tree, I don't think these names would be very clear in /usr/include. I'd rather use names in /usr/include that more clearly associate them with the kernel. Heck, even /usr/include/asm is inappropriate, but it's been there for so long that we really shouldn't try to get rid of it.

If I had it my way, we'd have, in the kernel tree, userland-aimed headers in include/linux/user and include/asm-<machine>/user, and have them installed in /usr/include/linux and /usr/include/asm-<machine>.

This means these headers shouldn't reference each other as linux/user/something.h, but rather as linux/something.h, such that they still work when installed in /usr/include/linux. This may require headers include/linux/something.h to include linux/user/something.h, but that's already part of the proposal.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#150 lore raw
In reply to Tonnerre
Quoted 16 lines · Tonnerre

Salut,

Quoted 4 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

Fnord alert: you're trying to change the API and the ABI of Linux. The thing
you fail to see is that Linux is not the only operating system in the world. Big companies are very prone of using their compatibility code to make their program run in any odd way they like without actually having to change
anything. Also, you're talking about assumptions that have been true for
decades.

Quoted 6 lines

Big companies are already eyeing Linux with fear because our kernels don't actually work and because we break a lot of compatibility in the stable series. If you now come up with something that breaks basically everything so they need a new libc and everything, it's rather likely that they'll run away.

I didn't say to change this now, not tomorrow...
Just tell them to change their code because, somewhere in the future, the API might change. Maybe the new kernel APIs are better, faster, more secure, so perhaps there is a good reason as why it would be better to use the new API.
And for those who want to use the old API, let's give them the ability, but also make it possible to compile the kernel with _only_ the new API. The different distributions will take care of it.

And about the compatibility layer.. the companies have to make a different one for each stable series anyways.

tom

#151 lore raw
In reply to Tonnerre
Quoted 1 line · Tonnerre

Fnord alert: you're trying to change the API and the ABI of Linux.

No he's not. He's changing the location where the API is defined, and the ABI doesn't change a bit.

Existing binaries will continue to run without modification.

Chris

#10 lore raw
In reply to David Howells
Quoted 2 lines · David Howells

How about calling the interface headers "kapi*/" instead of "user*/". In case you haven't guessed, "kapi" would be short for "kernel-api".

I don't think that change really makes any difference. The nomenclature really isn't _that_ important.

Signature · 1 line
dwmw2
#146 lore raw
In reply to Matthew Wilcox
Quoted 6 lines

This proposal doesn't address the asm-generic problem directly. Can I presume that you intend to also create linux/include/user-generic, install it as /usr/include/user-generic and create an asm-generic symlink that points to user-generic? A good problem file to be dealt with would be asm/errno.h

Agreed. I'd forgotten about that.

David

#18 lore raw
In reply to Alexandre Oliva
Quoted 14 lines · Alexandre Oliva
Quoted 11 lines · Matthew Wilcox
Quoted 5 lines · Alexandre Oliva

This means these headers shouldn't reference each other as linux/user/something.h, but rather as linux/something.h, such that they still work when installed in /usr/include/linux. This may require headers include/linux/something.h to include linux/user/something.h, but that's already part of the proposal.

That's going to take severe brain-ache to get right ... and worse, keep right. These headers aren't going to get tested outside the kernel tree often. So we'll have missing includes and files that only work if the <linux/> they're including is a kernel one rather than a user one.

Quoted 5 lines

How about moving the internals (i.e., what's not to be exported to userland) from linux and asm elsewhere, then?

Sure, it means significantly more churn in the kernel, but there's going to be a lot of moving stuff around one way or the other.

Not really. The kernel code was what was _allowed_ to be using those headers with those names in the first place; it was userspace which shouldn't necessarily have been doing so.

Quoted 5 lines

While at that, we could also split what's kernel internal for real and what's to be visible to external kernel modules as well. So we'd have 3 layers of headers, instead of two. I'm not sure this actually makes any sense though, since there might be lots of dependencies of headers for modules on internal headers.

All modules will be entirely dependent on the full set of kernel headers. Let's not go there.

Signature · 1 line
dwmw2
#152 lore raw
In reply to Tonnerre
Quoted 1 line · Tonnerre

Fnord alert: you're trying to change the API and the ABI of Linux. The thing

No we aren't. We're just trying to make the kernel interface more explicit. Nothing should change as far as userspace programs are concerned. It might need some coordination with the various libcs.

And enough with the doom-and-gloom big-companies-are-scared approach, OK? It's (a) untrue, (b) irrelevant, and (c) many of us work for big companies that are doing Linux already. We don't need to be lectured by you.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#17 lore raw
In reply to Matthew Wilcox
Quoted 11 lines · Matthew Wilcox
Quoted 5 lines · Alexandre Oliva

This means these headers shouldn't reference each other as linux/user/something.h, but rather as linux/something.h, such that they still work when installed in /usr/include/linux. This may require headers include/linux/something.h to include linux/user/something.h, but that's already part of the proposal.

That's going to take severe brain-ache to get right ... and worse, keep right. These headers aren't going to get tested outside the kernel tree often. So we'll have missing includes and files that only work if the <linux/> they're including is a kernel one rather than a user one.

How about moving the internals (i.e., what's not to be exported to userland) from linux and asm elsewhere, then?

Sure, it means significantly more churn in the kernel, but there's going to be a lot of moving stuff around one way or the other.

While at that, we could also split what's kernel internal for real and what's to be visible to external kernel modules as well. So we'd have 3 layers of headers, instead of two. I'm not sure this actually makes any sense though, since there might be lots of dependencies of headers for modules on internal headers.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#13 lore raw
In reply to David Woodhouse
Quoted 3 lines
Quoted 2 lines · Matthew Wilcox

Indeed. We could also make this transparent to userspace by using a script to copy the user-* headers to /usr/include. Something like this:

some administrator would like to only create symlinks for saving disk space.

others would like a true "install" with copying files so that they can delete the kernel sources anytime they do want.

for me i would install all those kernel realted files into the well known /lib/modules/<kernel-version>.
so its a no question that a single kernel install can be cleaned up by deleting those dir recursively.

the only part remaining somewhere else in the filesystem would then be /boot/bzImage since booting is a special case. there are several architectures where the bzImage is be able to reside in /lib/modules/... but there are others archs as well.

Quoted 7 lines · David Woodhouse
Quoted 3 lines · Matthew Wilcox

If we really wanted to get fancy, we could also sed __u32 to uint32_t. But that would probably cause more pain, confusion, hurt and bad feeling than I'd ever want to be responsible for.

Also true. Let's just use the standard types in the first place and not screw around with having to fix it up later.

uint32_t is the type you should get when using the standarized statement
#include <stdint.h>
Am i right? Does that header belong to the international C99 Standard? So you are suggesting the kernel interface should stick to that standard?

Changing it that way would hit a noticeable amount of non kernel code. Lets consider the opposite point of view. If we do postpone such a change to the standards for some 3 to 10 more years, then i expect the impact on software and problem for getting it solve is bigger.

Maybe that sort of change is something qualified for beeing done inside a true development kernel tree like Linux 2.7.xx which might then beeing qualified for getting a Linux 3.0.0 release kernel tree.

-Alex.

#153 lore raw
In reply to Matthew Wilcox
Quoted 8 lines · Matthew Wilcox
Quoted 1 line · Tonnerre

Fnord alert: you're trying to change the API and the ABI of Linux. The thing

No we aren't. We're just trying to make the kernel interface more explicit. Nothing should change as far as userspace programs are concerned. It might need some coordination with the various libcs.

Speaking of more explicit, there are various asm-ARCH header files that do or do not hide (via __KERNEL__) interfaces
such as: get_unaligned()
and the atomic operations.

So are these Linux kernel exported APIs, or do they belong in some library?

We'll need to decide on a consistent remedy for these, and such changes could be made at any time IMO (i.e., soon).

Quoted 3 lines

And enough with the doom-and-gloom big-companies-are-scared approach, OK? It's (a) untrue, (b) irrelevant, and (c) many of us work for big companies that are doing Linux already. We don't need to be lectured by you.

Signature · 1 line
~Randy
#19 lore raw
In reply to Alexandre Oliva
Quoted 2 lines · Alexandre Oliva

How about moving the internals (i.e., what's not to be exported to userland) from linux and asm elsewhere, then?

That's what I proposed several {months,weeks} ago. From a technical point of view it seems like a superior solution:

- /usr/include/{linux,asm} -> /usr/src/linux/include/{linux,asm} = no problems with kernel and userspace includes.
- while Linux is quite a big program, it is _one_ program = no need to patch the whole universe, only the kernel would need changing (of course broken userspace code would need fixing, but that's due to their brokeness and not due to the change).

Quoted 2 lines

Sure, it means significantly more churn in the kernel, but there's going to be a lot of moving stuff around one way or the other.

No doubt. Still, quite simple automatic operation, we've seen much larger patches.

API backward-compatibility (including C libs) is IMHO much more important than just some kernel patch size (I realize ABI compatibility would be preserved in all cases).

And we can do it anytime - no need for waiting for some libc change, no need for waiting for "2.7/2.8".

Quoted 2 lines

While at that, we could also split what's kernel internal for real and what's to be visible to external kernel modules as well.

Can't see benefits.

Signature · 1 line
Krzysztof Halasa
#154 lore raw
In reply to Randy.Dunlap
Quoted 7 lines · Randy.Dunlap

Speaking of more explicit, there are various asm-ARCH header files that do or do not hide (via __KERNEL__) interfaces
such as: get_unaligned()
and the atomic operations.

So are these Linux kernel exported APIs, or do they belong in some library?

Both of those are kernel-private and should not be visible.

Signature · 1 line
dwmw2
#147 lore raw
In reply to Matthew Wilcox
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines

(5) For userspace use (such as for glibc), the appropriate include/user*/

     directories should be selected and installed in /usr/include/ or wherever,
     and symlinks made. For example, on i386 arch boxes, you might find:

	SOURCE			INSTALLED AS
	======================	============
	include/user/		/usr/include/user/
	include/user-i386/	/usr/include/user-i386/
				/usr/include/linux -> user
				/usr/include/asm -> user-i386
Quoted 5 lines

This proposal doesn't address the asm-generic problem directly. Can I presume that you intend to also create linux/include/user-generic, install it as /usr/include/user-generic and create an asm-generic symlink that points to user-generic? A good problem file to be dealt with would be asm/errno.h

Do not allow asm-generic to be used direct by userspace.
The example with errno.h is handled by user-i386/errno.h which includes asm-generic/errno.h.

Would that do the trick - yes?

Sam

#168 lore raw
In reply to David Howells
Quoted 11 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

What we've come up with is this:

(1) Create new directories in the linux sources to shadow existing include directories:

Quoted 7 lines

NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/user/ include/linux/
include/user-*/ include/asm-*/

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

If we go for some resturcturing of include/ then we should get rid of the annoying asm symlink. Following layout deals with that:

include/<arch>/asm <= Files from include/asm-<arch>
include/<arch>/mach* <= Files from include/mach-*

This layout solve the symlink issue in an elegant way.
We need to do trivial changes to compiler options to make it work. Changing compiler options is much more relaible than using symlinks.

Then the userspace part would then be located in:
include/<arch>/user-asm

Userspace shall not be 'allowed' to include files from asm-generic direct, but asm-generic shall be part of the userspace api.

Quoted 8 lines

(2) Take each file from the shadowed directory. If it has any userspace

     relevant stuff, then:

     (b) Make kernel file #include the user file. So:

		[include/asm-i386/unistd.h]
		...
		#include <user-i386/unistd.h>

In example above it would become:
#include <user-asm/unistd.h>

With repsect to consistency of the userspace header files then we could define a few simple rules:
1) Files in include/user must be able to compile standalone - so they include all files they need.
2) They are not allowed to use any files outside the kernel
3) Must not use __KERNEL__ in any place

Writing a small script / Makefile that does these checks should be trivial. J�rn wrote something similar for include/linux at one point in time. If it is fast then we just let it be part of the standard compile so it is checked often.

Sam

#169 lore raw
Quoted 27 lines · Sam Ravnborg
Quoted 10 lines · David Howells

We've been discussing splitting the kernel headers into userspace API headers and kernel internal headers and deprecating the __KERNEL__ macro. This will permit a cleaner interface between the kernel and userspace; and one that's easier to keep up to date.

What we've come up with is this:

(1) Create new directories in the linux sources to shadow existing include directories:

Quoted 7 lines

NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/user/ include/linux/
include/user-*/ include/asm-*/

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

If we go for some resturcturing of include/ then we should get rid of the annoying asm symlink. Following layout deals with that:

include/<arch>/asm <= Files from include/asm-<arch>
include/<arch>/mach* <= Files from include/mach-*

Quoted 7 lines

This layout solve the symlink issue in an elegant way.
We need to do trivial changes to compiler options to make it work. Changing compiler options is much more relaible than using symlinks.

Then the userspace part would then be located in:
include/<arch>/user-asm

This complicates things for bi-arch architectures like x86_64 where one can use a dispatcher directory instead of a symlink to suit include/asm for 32bit as well as 64bit.

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#170 lore raw
Quoted 10 lines · Andreas Steinmetz
Quoted 10 lines

If we go for some resturcturing of include/ then we should get rid of the annoying asm symlink. Following layout deals with that:

include/<arch>/asm <= Files from include/asm-<arch>
include/<arch>/mach* <= Files from include/mach-*

This layout solve the symlink issue in an elegant way.
We need to do trivial changes to compiler options to make it work. Changing compiler options is much more relaible than using symlinks.

Quoted 7 lines
Quoted 3 lines

Then the userspace part would then be located in:
include/<arch>/user-asm

This complicates things for bi-arch architectures like x86_64 where one can use a dispatcher directory instead of a symlink to suit include/asm for 32bit as well as 64bit.

X86_64 does not create any special symlinks todays so I do not see the point? And still a symlink is just the wrong way to solve the problem. Adjusting options to the compiler is the way to do it.

Sam

#171 lore raw
Quoted 16 lines · Sam Ravnborg
Quoted 14 lines · Andreas Steinmetz
Quoted 13 lines

If we go for some resturcturing of include/ then we should get rid of the annoying asm symlink. Following layout deals with that:

include/<arch>/asm <= Files from include/asm-<arch>
include/<arch>/mach* <= Files from include/mach-*

This layout solve the symlink issue in an elegant way.
We need to do trivial changes to compiler options to make it work. Changing compiler options is much more relaible than using symlinks.

Then the userspace part would then be located in:
include/<arch>/user-asm

Quoted 5 lines
Quoted 1 line

This complicates things for bi-arch architectures like x86_64 where one can use a dispatcher directory instead of a symlink to suit include/asm for 32bit as well as 64bit.

X86_64 does not create any special symlinks todays so I do not see the point? And still a symlink is just the wrong way to solve the problem. Adjusting options to the compiler is the way to do it.

Ok, this is not in the kernel, but think of the following: an application needs and include/asm header. The application may be compiled as 32 bit and as 64 bit. Then you could use a real asm directory instead of a symlink and use, e.g. for asm/byteorder.h:

#ifdef __x86_64__
#include <asm-x86_64/byteorder.h>
#else
#include <asm-i386/byteorder.h>
#endif

This way both archs are satisfied from a userspace point of view without
having to take care if the source is compiled on 32bit or 64bit. If
you kill the symlink this neat dispatcher can't really be done anymore that easy.
It is not necessary to do anything in the kernel tree here except to keep the symlink to allow for this kind of userspace tweaks for certain architectures. This is especially important when there's need to use a include/linux header which itself needs an include/asm header as in this case with the
the little tweak always the right asm header is included.

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#155 lore raw
In reply to David Woodhouse
Quoted 10 lines · you
Quoted 7 lines · Randy.Dunlap

Speaking of more explicit, there are various asm-ARCH header files that do or do not hide (via __KERNEL__) interfaces such as:      get_unaligned()
and the atomic operations.

So are these Linux kernel exported APIs, or do they belong in some library?

Both of those are kernel-private and should not be visible.

The problem with these (atomic.h, bitops.h, byteorder.h, div64.h, list.h, spinlock.h, unaligned.h and xor.h) is that they provide functionality that is needed by many user application but not provided by the compiler or libc.

While I agree that it is an absolutely evil concept to include them from the kernel headers, we have to face that by not installing them, lots of this existing evil user code will be broken even more and someone has to pick up the pieces.

	Arnd <><
#156 lore raw
Quoted 9 lines · Arnd Bergmann

The problem with these (atomic.h, bitops.h, byteorder.h, div64.h, list.h, spinlock.h, unaligned.h and xor.h) is that they provide functionality that is needed by many user application but not provided by the compiler or libc.

While I agree that it is an absolutely evil concept to include them from the kernel headers, we have to face that by not installing them, lots of this existing evil user code will be broken even more and someone has to pick up the pieces.

The existing evil user code is broken already. Any of the above can be implemented in a such a way that it only works in the kernel anyway.

We don't need to preserve it any more than we need to preserve evil broken code which defines __KERNEL__ before including certain headers. We're supposed to be cleaning this crap up. If we want every evil broken hack that's currently possible to _remain_ possible, it's never going to work.

Signature · 1 line
dwmw2
#172 lore raw
In reply to Andreas Steinmetz
Quoted 11 lines · Andreas Steinmetz

Ok, this is not in the kernel, but think of the following: an application needs and include/asm header. The application may be compiled as 32 bit and as 64 bit. Then you could use a real asm directory instead of a symlink and use, e.g. for asm/byteorder.h:

#ifdef __x86_64__
#include <asm-x86_64/byteorder.h>
#else
#include <asm-i386/byteorder.h>
#endif

I think we can get rid of this hack when we move to split kernel headers. parisc, s390 and mips already have combined headers, and it should not be too hard to combine the user ABI headers for sparc, ppc and x86_64 as well, without having to merge the complete architecture and kernel header trees for them.

Building for ppc64 would then use something like 'gcc -Iinclude -Iinclude/ppc64 -Iinclude/user -Iinclude/user/ppc/' for a layout that has files in

include/linux/foo.h
include/$arch/asm/foo.h
include/generic/asm/foo.h

include/user/linux/foo.h           (installed as <linux/foo.h>)
include/user/$arch32/asm/foo.h     (installed as <asm/foo.h>
include/user/asm-generic/foo.h     (installed as <asm-generic/foo.h>

[Note how the first three become optional in this scheme. For files
that need an internal version an a user version, ppc64/asm/foo.h can use #include_next <asm/foo.h> to get to generic/asm/foo.h and user/ppc/asm/foo.h.]

For ia64, the headers should probably stay separate because I would expect them to differ more from i386/x86_64 than the other dual 32/64 architectures. I don't think that is a problem because you need separate tool chains for ia64 and i386 anyway, while the others can share the libc headers and the compiler backend already.

	Arnd <><
#157 lore raw
In reply to Arnd Bergmann
Quoted 20 lines · Arnd Bergmann
Quoted 11 lines · you
Quoted 7 lines · Randy.Dunlap

Speaking of more explicit, there are various asm-ARCH header files that do or do not hide (via __KERNEL__) interfaces
such as: get_unaligned()
and the atomic operations.

So are these Linux kernel exported APIs, or do they belong in some library?

Both of those are kernel-private and should not be visible.

The problem with these (atomic.h, bitops.h, byteorder.h, div64.h, list.h, spinlock.h, unaligned.h and xor.h) is that they provide functionality that is needed by many user application but not provided by the compiler or libc.

Quoted 4 lines

While I agree that it is an absolutely evil concept to include them from the kernel headers, we have to face that by not installing them, lots of this existing evil user code will be broken even more and someone has to pick up the pieces.

That's addressing a different problem. I agree with David W. that we need to clean the kernel headers up. Let libc or libxyz provide the missing functionality. The borken programs were stealing something that wasn't promised to them AFAIK.

IOW, let's decide the right thing to do and implement it.

Signature · 1 line
~Randy
#158 lore raw
Quoted 5 lines · Randy.Dunlap

That's addressing a different problem. I agree with David W. that we need to clean the kernel headers up. Let libc or libxyz provide the missing functionality. The borken programs were stealing something that wasn't promised to them AFAIK.

Not only wasn't it promised; it wasn't even working on some architectures anyway.

Signature · 1 line
dwmw2
#159 lore raw
Quoted 10 lines · David Woodhouse
Quoted 5 lines · Randy.Dunlap

That's addressing a different problem. I agree with David W. that we need to clean the kernel headers up. Let libc or libxyz provide the missing functionality. The borken programs were stealing something that wasn't promised to them AFAIK.

Not only wasn't it promised; it wasn't even working on some architectures anyway.

Well, at least on s390 some effort was spent on making it work (see e.g. asm-s390/spinlock.h). I don't really mind removing that functionality, but I'd prefer still installing some files into /usr/include/asm/spinlock.h et.al that contain an #error and a pointer to an explanation, so users can complain to the right people (i.e. not me ;-).

	Arnd <><
#160 lore raw
In reply to Arnd Bergmann
Quoted 2 lines

The problem with these (atomic.h

that is a very non portable header and there are several good alternatives (see the apr library for example). In fact. atomic.h is *dangerous* in userspace, it is only atomic if CONFIG_SMP is set, so if you compile your app on a machine without that set and then run it on an smp machine, you are not atomic.

Quoted 2 lines

, bitops.h

again not portable

Quoted 2 lines

, byteorder.h,

there are perfectly good alternatives in glibc

Quoted 2 lines

div64.h,

huh? what is wrong with "/" in C

Quoted 1 line

list.h

this one I can see

Quoted 2 lines

, spinlock.h

EHHHH????? Spinlocks in userland? You got to be kidding.

Quoted 1 line

, unaligned.h

weird

Quoted 1 line

and xor.h)

xor.h is very raid specific (and GPL with lots of code, so a license trap)

#167 lore raw
In reply to Arnd Bergmann
Quoted 4 lines

The problem with these (atomic.h, bitops.h, byteorder.h, div64.h, list.h, spinlock.h, unaligned.h and xor.h) is that they provide functionality that is needed by many user application but not provided by the compiler or libc.

It's already broken on many architectures and just happens to work on some.

#161 lore raw
In reply to Arjan van de Ven

Ok, I've looked for places where someone actually tried using the kernel headers by googling for /usr/include/asm/foo.h. The good news is that marking these files broken in glibc-kernheaders has already pointed most authors to the source of the problem.

Quoted 8 lines · Arjan van de Ven
Quoted 2 lines

The problem with these (atomic.h

that is a very non portable header and there are several good alternatives (see the apr library for example). In fact. atomic.h is *dangerous* in userspace, it is only atomic if CONFIG_SMP is set, so if you compile your app on a machine without that set and then run it on an smp machine, you are not atomic.

Yes, it appears that most people who attempted to use this have already been bitten so hard that they stopped trying.

Quoted 4 lines
Quoted 2 lines

, bitops.h

again not portable

netbase-ping6 tried to use it, along with some other applications that I had not heard of.

Quoted 4 lines
Quoted 2 lines

, byteorder.h,

there are perfectly good alternatives in glibc

Google found abuses of byteorder.h in kdeedu, dbootstrap and netatalk, I would expect to find many more if I kept looking.

Quoted 4 lines
Quoted 2 lines

div64.h,

huh? what is wrong with "/" in C

Ok, probably nobody has tried to do funny things with this one.

Quoted 3 lines
Quoted 1 line

list.h

this one I can see

Surprisingly few abuses. It is already been protected with #ifdef __KERNEL__ for some time.

Quoted 4 lines
Quoted 2 lines

, spinlock.h

EHHHH????? Spinlocks in userland? You got to be kidding.

I don't think it's that uncommon to use spinlocks in user space, IIRC samba (tdb) and some databases implement their own version of user spinlocks. The kernel implementation is used at least in "Open Runtime Platform" and in "Chrony". Chrony even has an FAQ entry on how to circumvent the "#ifndef __KERNEL__" that was added in Red Hat Linux...

Quoted 3 lines
Quoted 1 line

, unaligned.h

weird

At least one application that I use every day (vdr) tried including asm/unaligned.h, but it's rather uncommon otherwise.

Quoted 4 lines
Quoted 1 line

and xor.h)

xor.h is very raid specific (and GPL with lots of code, so a license trap)

Right, this one has never been used from user space AFAICS.

	Arnd <><
#162 lore raw
Quoted 11 lines
Quoted 8 lines

The problem with these (atomic.h

that is a very non portable header and there are several good alternatives (see the apr library for example). In fact. atomic.h is *dangerous* in userspace, it is only atomic if CONFIG_SMP is set, so if you compile your app on a machine without that set and then run it on an smp machine, you are not atomic.

Yes, it appears that most people who attempted to use this have already been bitten so hard that they stopped trying.

and for people who'd be tempted for new code it should at least warn if not error even right now

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

, byteorder.h,

there are perfectly good alternatives in glibc

Google found abuses of byteorder.h in kdeedu, dbootstrap and netatalk, I would expect to find many more if I kept looking.

it's still very much abuse, and quite nasty for dual endian platforms ;)

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

, spinlock.h

EHHHH????? Spinlocks in userland? You got to be kidding.

I don't think it's that uncommon to use spinlocks in user space, IIRC samba (tdb) and some databases implement their own version of user spinlocks.

implementing your own is still evil, but glibc provides similar constructs already. busy waiting in userspace is evil anyway (use futex :) and.... just as with atomic.h, spinlock.h only gets compiled in for CONFIG_SMP so the same caveats apply

Quoted 4 lines

The kernel implementation
is used at least in "Open Runtime Platform" and in "Chrony". Chrony even has an FAQ entry on how to circumvent the "#ifndef __KERNEL__" that was added in Red Hat Linux...

Chrony then needs to make their own (by copying and cleaning up for all I care).

#164 lore raw
In reply to Arnd Bergmann
Quoted 5 lines · Previously Arnd Bergmann

Ok, I've looked for places where someone actually tried using the kernel headers by googling for /usr/include/asm/foo.h. The good news is that marking these files broken in glibc-kernheaders has already pointed most authors to the source of the problem.

If you want a challenge try if you can get strace to compile with glibc-kernheaders.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#165 lore raw
Quoted 9 lines · Wichert Akkerman
Quoted 5 lines · Previously Arnd Bergmann

Ok, I've looked for places where someone actually tried using the kernel headers by googling for /usr/include/asm/foo.h. The good news is that marking these files broken in glibc-kernheaders has already pointed most authors to the source of the problem.

If you want a challenge try if you can get strace to compile with glibc-kernheaders.

the rh strace package seems to cope....

and while glibc-kernheaders hides the kernel internal structures, it does expose all the external ones...

#173 lore raw
In reply to Arnd Bergmann
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines

#ifdef __x86_64__
#include <asm-x86_64/byteorder.h>
#else
#include <asm-i386/byteorder.h>
#endif

Quoted 6 lines

I think we can get rid of this hack when we move to split kernel headers. parisc, s390 and mips already have combined headers, and it should not be too hard to combine the user ABI headers for sparc, ppc and x86_64 as well, without having to merge the complete architecture and kernel header trees for them.

If you can produce merged x86/x86_64 ABI headers this would be a nice solution.

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#174 lore raw
Quoted 9 lines · Andreas Steinmetz
Quoted 6 lines

I think we can get rid of this hack when we move to split kernel headers. parisc, s390 and mips already have combined headers, and it should not be too hard to combine the user ABI headers for sparc, ppc and x86_64 as well, without having to merge the complete architecture and kernel header trees for them.

If you can produce merged x86/x86_64 ABI headers this would be a nice solution.

I can do that (and the same for ppc/ppc64) as soon as we have
a) A patch that first splits out the ABI headers for i386
b) persuaded ak that he will look at what I come up with

	Arnd <><
#166 lore raw
In reply to Arjan van de Ven
Quoted 3 lines · Arjan van de Ven
Quoted 1 line

, spinlock.h

EHHHH????? Spinlocks in userland? You got to be kidding.

There is pthread_spin_{init,lock,unlock,trylock,destroy} in libpthread.so.

Jakub

#163 lore raw
In reply to Arjan van de Ven
Quoted 4 lines · Arjan van de Ven

implementing your own is still evil, but glibc provides similar constructs already. busy waiting in userspace is evil anyway (use futex :) and.... just as with atomic.h, spinlock.h only gets compiled in for CONFIG_SMP so the same caveats apply

Yeah, this is on it's way out now that futexes exist, but KDE (for one) used to use spinlocks for a thread-pooled malloc (spin-sleep-spin-sleep though, calling either sched_yield or nanosleep if the lock-grab failed - so it wasn't a pure busy-wait). It didn't reuse spinlock.h from the kernel, but it did use the general concept, since the old pre-futex posix mutexes were crazy-slow for a lock which was heavily used but rarely contended.

#30 lore raw
In reply to Greg KH
Quoted 4 lines · Greg KH

The justification is that it doesn't properly describe the variable size (think userspace 32 bit variable vs. kernelspace 32 bit variable.) We need to stick with the proper __u32 type variables for data that crosses the userspace/kernelspace boundry.

uint32_t is defined to be exactly 32 bits wide, so where's the problem in using it instead of __u32 in the headers that describe the user/kernel interface? (Ditto for uint{8,16,64}_t, of course.)

Paul.

#31 lore raw
Quoted 4 lines · Paul Mackerras

uint32_t is defined to be exactly 32 bits wide, so where's the problem in using it instead of __u32 in the headers that describe the user/kernel interface? (Ditto for uint{8,16,64}_t, of course.

Ok, this discussion has gone on for too long anyway, but let's make it easier for everybody. The kernel uses u8/u16/u32 because:

- the kernel should not depend on, or pollute user-space naming. YOU MUST NOT USE "uint32_t" when that may not be defined, and user-space rules for when it is defined are arcane and totally arbitrary.

- since the kernel cannot use those types for anything that is visible to user space anyway, there has to be alternate names. The tradition is to prepend two underscores, so the kernel would have to use "__uint32_t" etc for its header files.

- at that point, there's no longer any valid argument that it's a "standard type" (it ain't), and I personally find it a lot more readable to just use the types that the kernel has always used:
__u8/__u16/__u32. For stuff that is only used for the kernel, the shorter "u8/u16/u32" versions may be used.

In short: having the kernel use the same names as user space is ACTIVELY BAD, exactly because those names have standards-defined visibility, which means that the kernel _cannot_ use them in all places anyway. So don't even _try_.

On the bigger question of what to do with kernel headers in general, let's just make one thing clear: the kernel headers are for the kernel, and big and painful re-organizations that don't help _existing_ user space are not going to happen.

In particular, any re-organization that breaks _existing_ uses is totally pointless. If you break existing uses, you might as well _not_ re- organize, since if you consider kernel headers to be purely kernel- internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

So I think this whole discussion has been largely pointless. We undeniably have existing users of kernel headers. That's just a fact. If we break them, it doesn't _matter_ how the kernel headers look, and then "existing practice" is about as good an organization as anything, and breaking things just to break things is not something I'm in the least interested in. "Beauty" comes secondary to "usefulness".

So I would encourage people to think about ways to clean up some of the worst warts, but take into account:
- testing it out with whatever random collection of old distributions and special applications is almost impossible. So every single step of the way should be (a) small and (b) obviously not break any current users.
- cleanup just for the sake of cleanup always needs to take pain into account. If you cannot make each small step "worth it", then just don't do it. If the "cleanup" just adds another file and doesn't actually _help_ anything that you can point to, it's not a cleanup, it's just an exercise in wanking.

IOW, I seriously doubt any "let's reorganize header files just to make it look good" _ever_ accomplished anything. But if there are _specific_ header files that have _specific_ problems, let's look at maybe solving those. And if you cannot point to a specific problem with a suggested specific solution, please don't cc me.

Linus

#32 lore raw
Quoted 8 lines · Linus Torvalds

In short: having the kernel use the same names as user space is ACTIVELY BAD, exactly because those names have standards-defined visibility, which means that the kernel _cannot_ use them in all places anyway. So don't even _try_.

On the bigger question of what to do with kernel headers in general, let's just make one thing clear: the kernel headers are for the kernel, and big and painful re-organizations that don't help _existing_ user space are not going to happen.

In particular, any re-organization that breaks _existing_ uses is totally
pointless. If you break existing uses, you might as well _not_ re-
organize, since if you consider kernel headers to be purely kernel-
internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

I don't think any drastic reorganization is even necessary.

Mariusz Mazur <[email protected]> updates http://ep09.pld-linux.org/~mmazur/linux-libc-headers/ for each 2.6.x kernel release. linux-libc-headers are the kernel headers, with all the kernel-specific stuff stripped out. i.e. userland ABI only. Not sure how many distros have started picking that up yet... I think Arjan said Fedora Core had, or would.

If people want to go beyond that, IMHO it would be simple and easy to start putting new kernel headers in include/kernel (or somesuch). That way there are no massive reorganizations; kernel-specific stuff gets slowly migrated to a kernel-specific area.

Jeff

#33 lore raw
Quoted 4 lines · Jeff Garzik

If people want to go beyond that, IMHO it would be simple and easy to start putting new kernel headers in include/kernel (or somesuch). That way there are no massive reorganizations; kernel-specific stuff gets slowly migrated to a kernel-specific area.

ITYM "to areas where it actually gets used". A _lot_ in include/* is used only by a couple of drivers and should've been sitting in drivers/*/* instead.

#34 lore raw
Quoted 3 lines · Al Viro

ITYM "to areas where it actually gets used". A _lot_ in include/* is used only by a couple of drivers and should've been sitting in drivers/*/* instead.

Amen to that! Just because it is a header doesn't mean it belongs in include. Just because it is a #define or a struct definition does not mean it belongs in a header.

So many people don't get the idea of exposed vs private interfaces.

#35 lore raw
In reply to Jeff Garzik
Quoted 8 lines · Jeff Garzik
Quoted 5 lines

In particular, any re-organization that breaks _existing_ uses is totally pointless. If you break existing uses, you might as well _not_ re- organize, since if you consider kernel headers to be purely kernel- internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

I don't think any drastic reorganization is even necessary.

Well, we want things to be split. However a split doesn't mean a reorganisation as far as userspace visibility is concerned; the filenames can still be the same, and the typenames etc etc, you just do the kernel internal additions in a different dir/header

Not breaking userland ever is a dream only, anytime we touch any header, some userland breaks (they use our spinlock code even!)

Quoted 3 lines

kernel-specific stuff stripped out. i.e. userland ABI only. Not sure how many distros have started picking that up yet... I think Arjan said Fedora Core had, or would.

nope.

Fedora has it's own set of cleaned up headers. Cleaned up in the sense that tehre's no #ifdef KERNEL anymore, no inlines (because that's a license trap, and in addition, all inlines were kernel specific anyway) and all structs which had spinlocks / semaphores and other kernel private structures in are removed as well (because they are clearly kernel internal).

#42 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

In particular, any re-organization that breaks _existing_ uses is totally pointless. If you break existing uses, you might as well _not_ re- organize, since if you consider kernel headers to be purely kernel- internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

Agreed, with the proviso that the inclusion of bitops.h, spinlock.h, and other kernel-private stuff like that should not be considered 'existing uses'. It's _already_ broken. It won't compile on some architectures, and will silently do the wrong thing on others. Causing the compile to fail consistently would be a _feature_.

Quoted 6 lines

So I think this whole discussion has been largely pointless. We undeniably have existing users of kernel headers. That's just a fact. If we break them, it doesn't _matter_ how the kernel headers look, and then "existing practice" is about as good an organization as anything, and breaking things just to break things is not something I'm in the least interested in. "Beauty" comes secondary to "usefulness".

Usefulness is what we're after. Preventing the naïve user from including spinlock.h and thinking that it'll give useful spinlocks would be useful.

I've lost track of the number of times things have broken because of incorrect use of kernel headers from userspace. That's what we're trying to fix -- by putting only the bits which are _supposed_ to be visible into files which userspace sees, where we know they define part of the userspace API and hence we can be extremely careful when editing them.

I don't think it makes sense at this point for us to bury our collective heads in the sand and pretend there isn't a problem here that's worth fixing.

I agree that it should be obviously correct though -- and that's why we're trying to end up with a structure that in the first pass would give us in userspace essentially what we already have in the various glibc-kernheaders packages, but without the constant and unnecessary need for some poor sod to keep those up to date by hand.

Signature · 1 line
dwmw2
#47 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

In particular, any re-organization that breaks _existing_ uses is totally pointless. If you break existing uses, you might as well _not_ re- organize, since if you consider kernel headers to be purely kernel- internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

Recently I had some users complaining because their userspace program that includes <asm/atomic.h> compiles OK as a ppc64 program but not as a ppc32 program. The reason for that is that asm-ppc/atomic.h has #ifdef __KERNEL__ around the inline definitions of atomic_inc et al., but asm-ppc64/atomic.h doesn't.

My response was that they shouldn't be using <asm/atomic.h> because
(a) it's an internal kernel header, not part of the kernel ABI, (b) it's not portable (it happens to work on some popular architectures, but won't work on all) and (c) they're really only entitled to use those particular inline function definitions in GPL'd programs anyway.

To which the obvious reply is "well then why is it in /usr/include?" It's because the glibc build process copies all the kernel headers to /usr/include, but that's not an answer from a user's point of view.

I'd be interested to hear your take on this. Should we try to make our atomics easy and safe for userspace to use (including putting them under the LGPL)? Or can you see a better solution?

Paul.

#48 lore raw
Quoted 14 lines · Paul Mackerras
Quoted 5 lines · Linus Torvalds

In particular, any re-organization that breaks _existing_ uses is totally pointless. If you break existing uses, you might as well _not_ re- organize, since if you consider kernel headers to be purely kernel- internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

Recently I had some users complaining because their userspace program that includes <asm/atomic.h> compiles OK as a ppc64 program but not as a ppc32 program. The reason for that is that asm-ppc/atomic.h has #ifdef __KERNEL__ around the inline definitions of atomic_inc et al., but asm-ppc64/atomic.h doesn't.

Quoted 5 lines

My response was that they shouldn't be using <asm/atomic.h> because
(a) it's an internal kernel header, not part of the kernel ABI, (b) it's not portable (it happens to work on some popular architectures, but won't work on all) and (c) they're really only entitled to use those particular inline function definitions in GPL'd programs anyway.

and (d) on x86 at least the "atomic" part of the atomic ops are #ifdef CONFIG_SMP, which might well not be set in userland... so you get a false sense of atomicity.

Quoted 3 lines

I'd be interested to hear your take on this. Should we try to make our atomics easy and safe for userspace to use (including putting them under the LGPL)? Or can you see a better solution?

it's not the kernel's job to provide this functionality imo. glibc could. glib does. apr does. there's several options already. apr is BSD licensed for exanpmle, glib LGPL.

#49 lore raw
Quoted 5 lines · Arjan van de Ven
Quoted 3 lines

I'd be interested to hear your take on this. Should we try to make our atomics easy and safe for userspace to use (including putting them under the LGPL)? Or can you see a better solution?

it's not the kernel's job to provide this functionality imo.

Sometimes the functionality would have to be entirely reimplemented to make it viable for userspace. For FR-V we set aside a condition register for use in the kernel, so we can emulate ll/sc with multiple-issue instructions. That can never work in userspace.

We can't just pretend there isn't a problem here.

Signature · 1 line
dwmw2
#26 lore raw
In reply to Grzegorz Kulewski

Grzegorz Kulewski <[email protected]>:

Quoted 8 lines
Quoted 1 line

(b) Make kernel file #include the user file.

Does kernel really need to include user headers? When it is definition of some const then it should be defined in one file (to be sure it has only one definition). But user headers may have some compatibility hacks that kernel do not need (and even maybe does not want) to have.

How you will handle that?

I must have mis-explained it. I meant:

Quoted 1 line
Quoted 1 line

(b) Make kernel file #include the user/ file.

I don't actually mean that the kernel should in anyway touch userspace headers. I was referring to the bits split out into user-xxxx/ directories.

David

#50 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

So I would encourage people to think about ways to clean up some of the worst warts, but take into account:
- testing it out with whatever random collection of old distributions and special applications is almost impossible. So every single step of the way should be (a) small and (b) obviously not break any current users.

If I understand correctly the above means "include/* userland breakage problems open for fixing" and the "kernel includes aren't for userland" part is obsolete?

Quoted 11 lines

- cleanup just for the sake of cleanup always needs to take pain into account. If you cannot make each small step "worth it", then just don't do it. If the "cleanup" just adds another file and doesn't actually _help_ anything that you can point to, it's not a cleanup, it's just an exercise in wanking.

IOW, I seriously doubt any "let's reorganize header files just to make it look good" _ever_ accomplished anything. But if there are _specific_ header files that have _specific_ problems, let's look at maybe solving those. And if you cannot point to a specific problem with a suggested specific solution, please don't cc me.

Potential breakage is mostly a non issue - since vanilla kernel headers are mostly useless for userland anyway, one must either patch them or use my headers (fedora's?). As for my headers - I can always provide a compatibility layer that will look like the old tree but in fact use the new tree.* And that also means application specific hacks not to break some app (I'm doing that now anyway, so no difference).

I'm *really* for fixing the header problem once and for all one way (lifting the 'kernel includes aren't for userland' ban) or the other (making a separate, userland available part that glibc and other apps can use cleanly). Maintaining a separate tree of those headers ain't much fun and it would be nice if I could quit doing that some time in the future.

* Fluent and (mostly) painless migration guaranteed or your money back.

Signature · 5 lines
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
#16 lore raw
In reply to David Howells
Quoted 2 lines · David Howells

How about calling the interface headers "kapi*/" instead of "user*/". In case you haven't guessed, "kapi" would be short for "kernel-api".

I've seen kapi being used to reference the api exposed by the kernel to modules, but not the abi exposed to userland. ukabi sounds more appropriate for the latter, although many people might wonder what's with this United Kingdom ABI :-)

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#51 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

On the bigger question of what to do with kernel headers in general, let's just make one thing clear: the kernel headers are for the kernel, and big and painful re-organizations that don't help _existing_ user space are not going to happen.

So, of the following two options, which one you think everybody should pick?

- Linux gets to define the ABI between kernel and userland, and userland must duplicate the contents of headers in which the kernel defines the kernel<->userland ABI, tracking changes in them in the hope that nothing falls through the cracks

- we move the kernel<->userland to a separate package, where it's maintained such that it can be used by both kernel and userland, and Linux will only build when given a pointer to the location of this package.

I don't think you can have it both ways. As long as the kernel wants freedom to set the ABI, either let people use the headers where you define it, or maintain the ABI separately and let people use them. Forcing others to maintain duplicates of your headers will just get most people to use your headers for purposes they weren't meant to be used; maintaining two similar but far from identical sets of headers in sync is a major pain, that's why few people do it, and even fewer people do it properly.

Refusing to help in the effort to maintain them for wishes of freedom to tinker or allergy to external dependencies haven't worked as excuses in the past, and I think they never will. Duplicating the effort is just stupid, and requiring others to do so is arrogant. Why shouldn't we just get together and have something that works for both?

Or how about we turn the table around and define an ABI that the kernel must implement, and then have the kernel have to keep track of the changes in such external ABI definition, instead of the way things (fail to) work now? How comfortable would you feel with this arrangement?

My usual reply when I hear this argument that the kernel headers that define the ABI with userland are to be used by the kernel only, and someone else is supposed to create the headers for the same purpose to be used by userland, is that you generally won't find the headers that expose the interface of a library in a separate package (think libz.{a,so} and zlib.h). They're part of the same package, built out of the same sources, and there's a reason for that: if they get out of sync, it won't serve its purpose. It makes far more sense to maintain them together.

You won't find zlib.h in packages that just want to use zlib, unless they contain zlib in its entirety, just in case it's not pre-installed. I think this makes perfect sense. Requiring libc or a separate package used by libc to contain copies of headers that define the API/ABI exported by the kernel to userland is just the same sort of nonsense, with the difference that it's a way bigger set of headers we're talking about, which renders the problem even worse.

Sure you can take the easy way out and claim you told people they shouldn't do what they're doing when they run into problems because of including kernel headers from userland. But as soon as you reject proposals to clean things up such that we would have a set of headers that both kernel and userland will be able to use, in a way that would have the headers still conveniently (for you) sitting in the kernel source tree, I can't see that you could say it's not your problem with a straight face any more.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#52 lore raw
Quoted 8 lines · Alexandre Oliva
Quoted 5 lines

On the bigger question of what to do with kernel headers in general, let's just make one thing clear: the kernel headers are for the kernel, and big and painful re-organizations that don't help _existing_ user space are not going to happen.

So, of the following two options, which one you think everybody should pick?

Ok, that's pretty straightforward:

Quoted 4 lines

- Linux gets to define the ABI between kernel and userland, and userland must duplicate the contents of headers in which the kernel defines the kernel<->userland ABI, tracking changes in them in the hope that nothing falls through the cracks

This is unquestionably true. The kernel obviously _does_ define the ABI, and userland just lives with it. At some point you have to track things, just because new features etc just can't be sanely handled any other way.

That said, I think we can make the tracking _easier_.

Quoted 4 lines

- we move the kernel<->userland to a separate package, where it's maintained such that it can be used by both kernel and userland, and Linux will only build when given a pointer to the location of this package.

No. Quite frankly, I absolutely hate projects that do that. I gave up on following several interesting projects (mostly media decoding) just because it was just too damn painful to try to make sure that five different packages were all in sync, and you couldn't find one site to just download it all from.

So I want the kernel to be as stand-alone as humanly possible. There are a few dependencies I can't avoid, notably the compiler, and there the kernel triest to be as permissive as humanly reasonable, and we've always tried to keep the required tools down to a manageable level (ie even when people were clamorign for graphical configurations etc, the kernel always made that very optional indeed).

***

Now, on trying to make the tracking _easier_, I would not mind at all to move (well-defined) things around a bit to make it clearer what is actually exported to user space. But on the other hand, I don't think it's actively wrong either to just "mark" them in the headers some way, and have tools to extract it automatically _without_ having to separate them into some magic location.

In fact, in many ways I'd prefer to have source-level annotations like "this is exported to user space" over trying to gather things in one place. It would need to have some automated checking ability (that probably most people wouldn't run, and would break every once in a while, but hey, that's nothing new. That's how "checkconfig" etc works).

But it doesn't have to be "one or the other". I think we could annotate some things that are nasty to export (and quite frankly, it's not like this is something "new"; __KERNEL__ is really nothing but a stupid sort of an "annotation" already, except it's likely the wrong way around: we should not annotate the stuff that is kernel-only, we should annotate the stuff that is user-visible, since that should be the exception rather than the rule).

But even if we decide to have part of the ABI annotated, part of it could be split out better. For example, I think the "posix_types.h" split ended up working really quite well, and it cleaned up a _huge_ number of type issues that people have probably already forgotten because that split was done so long ago.

So we _have_ had great success with the "split out specific things" before, and I think we should do it whenever there is an obvious chance, and something is abstracted out and well-defined enough that we can do so.

But in general, I do kind of like the explicit marking. The same way we explicitly mark the functions inside the kernel that we expose to modules, we could try to mark the data structures and values that we expose to user space. That tends to "work".

Linus

#175 lore raw
In reply to David Howells
Followup to:  <[email protected]>
By author:    David Howells <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 6 lines

What we've come up with is this:

(1) Create new directories in the linux sources to shadow existing include directories:

Quoted 8 lines

NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/user/ include/linux/
include/user-*/ include/asm-*/

Note that this doesn't take account of the other directories under include/, but I don't think they're relevant.

I'm not sure if user is a good choice, and user-* is even worse. Most people seem to have suggested include/linux-abi for this; I would personally prefer include/linux-abi/arch for the second.

Quoted 7 lines

(2) Take each file from the shadowed directory. If it has any userspace relevant stuff, then:

(a) Transfer this stuff into a file of the same name in the new directory. So, for example, the syscall number list from include/asm-i386/unistd.h will be transferred to include/user-i386/unistd.h.

I'm not sure you can do such a 1:1 mapping. In fact, there are cases where you definitely don't want to, because the current structure doesn't make much sense.

Quoted 7 lines
     (b) Make kernel file #include the user file. So:

		[include/asm-i386/unistd.h]
		...
		#include <user-i386/unistd.h>
		...

Good...

Quoted 8 lines
     (c) Where a user header file requires something from another header file
	 (such as a type), that file should include a suitable user header file
	 directly:

		[include/user-i386/termio.h]
		...
		#include <user/types.h>
		...

Good...

Quoted 9 lines
     (d) stdint types should be used where possible.

		[include/user-i386/termios.h]
		struct winsize {
			uint16_t ws_row;
			uint16_t ws_col;
			uint16_t ws_xpixel;
			uint16_t ws_ypixel;
		};

Good, except your "struct winsize" is bad; you're stepping on namespace which belongs to userspace. Since we can't use typedefs on struct tags, I suggest:

struct __kstruct_winsize {
       /* ... */
};

.. and userspace can do:

#define __kstruct_winsize winsize

Quoted 7 lines
     (e) These header files should be bounded with __USER_XXXXX_H conditionals:

		[include/user-i386/termios.h]
		#ifndef __USER_I386_TERMIOS_H
		#define __USER_I386_TERMIOS_H
		...
		#endif /*  __USER_I386_TERMIOS_H */

Good...

Quoted 3 lines

(3) Remove all #if(n)def __KERNEL__ clauses.

(4) Remove the -D__KERNEL__ from the master kernel Makefile.

Bad! There is code in the kernel which can compile in userspace for testing. This is highly valuable and should be kept.

-hpa

#53 lore raw
In reply to Linus Torvalds

Speaking of include/* cleanups, here's a work of about 10 minutes
(and yes, result does build):

* private stuff moved into fs/affs/affs.h
* include/linux/affs_fs_{i,sb}.h removed, since they do not contain
anything that might be used outside of fs/affs/*
* includes in fs/affs/* trimmed (as the matter of fact, only super.c
needed something not pulled by affs.h)
* duplicate MODULE_LICENSE removed from fs/affs/inode.c (fs/affs/super.c
already has exact same thing)
* bogus extern (affs_fs()) removed (used to be in affs_fs.h, function
in question simply doesn't exist)

That leaves affs_fs.h containing only AFFS_SUPER_MAGIC declaration and amigaffs.h also seriously cleaned up; it still has a bunch of defines that are almost certainly never used in userland (they refer to ->b_data and in-core affs superblock), but these might in theory be useful (similar stuff for ext2 *is* used by userland code).

If Roman is OK with their removal, AFFS_DATA and friends should also get moved to fs/affs/affs.h.

----

fs/affs/affs.h +283 -0
diff -urN RC10-rc2-bk12-base/fs/affs/affs.h RC10-rc2-bk12-current/fs/affs/affs.h
@@ -0,0 +1,283 @@
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/buffer_head.h>
+#include <linux/affs_fs.h>
+#include <linux/amigaffs.h>
+
+#define AFFS_CACHE_SIZE		PAGE_SIZE
+
+#define AFFS_MAX_PREALLOC	32
+#define AFFS_LC_SIZE		(AFFS_CACHE_SIZE/sizeof(u32)/2)
+#define AFFS_AC_SIZE		(AFFS_CACHE_SIZE/sizeof(struct affs_ext_key)/2)
+#define AFFS_AC_MASK		(AFFS_AC_SIZE-1)
+
+struct affs_ext_key {
+	u32	ext;				/* idx of the extended block */
+	u32	key;				/* block number */
+};
+
+/*
+ * affs fs inode data in memory
+ */
+struct affs_inode_info {
+	u32	 i_opencnt;
+	struct semaphore i_link_lock;		/* Protects internal inode access. */
+	struct semaphore i_ext_lock;		/* Protects internal inode access. */
+#define i_hash_lock i_ext_lock
+	u32	 i_blkcnt;			/* block count */
+	u32	 i_extcnt;			/* extended block count */
+	u32	*i_lc;				/* linear cache of extended blocks */
+	u32	 i_lc_size;
+	u32	 i_lc_shift;
+	u32	 i_lc_mask;
+	struct affs_ext_key *i_ac;		/* associative cache of extended blocks */
+	u32	 i_ext_last;			/* last accessed extended block */
+	struct buffer_head *i_ext_bh;		/* bh of last extended block */
+	loff_t	 mmu_private;
+	u32	 i_protect;			/* unused attribute bits */
+	u32	 i_lastalloc;			/* last allocated block */
+	int	 i_pa_cnt;			/* number of preallocated blocks */
+#if 0
+	s32	 i_original;			/* if != 0, this is the key of the original */
+	u32	 i_data[AFFS_MAX_PREALLOC];	/* preallocated blocks */
+	int	 i_cache_users;			/* Cache cannot be freed while > 0 */
+	unsigned char i_hlink;			/* This is a fake */
+	unsigned char i_pad;
+	s32	 i_parent;			/* parent ino */
+#endif
+	struct inode vfs_inode;
+};
+
+/* short cut to get to the affs specific inode data */
+static inline struct affs_inode_info *AFFS_I(struct inode *inode)
+{
+	return list_entry(inode, struct affs_inode_info, vfs_inode);
+}
+
+/*
+ * super-block data in memory
+ *
+ * Block numbers are adjusted for their actual size
+ *
+ */
+
+struct affs_bm_info {
+	u32 bm_key;			/* Disk block number */
+	u32 bm_free;			/* Free blocks in here */
+};
+
+struct affs_sb_info {
+	int s_partition_size;		/* Partition size in blocks. */
+	int s_reserved;			/* Number of reserved blocks. */
+	//u32 s_blksize;			/* Initial device blksize */
+	u32 s_data_blksize;		/* size of the data block w/o header */
+	u32 s_root_block;		/* FFS root block number. */
+	int s_hashsize;			/* Size of hash table. */
+	unsigned long s_flags;		/* See below. */
+	uid_t s_uid;			/* uid to override */
+	gid_t s_gid;			/* gid to override */
+	umode_t s_mode;			/* mode to override */
+	struct buffer_head *s_root_bh;	/* Cached root block. */
+	struct semaphore s_bmlock;	/* Protects bitmap access. */
+	struct affs_bm_info *s_bitmap;	/* Bitmap infos. */
+	u32 s_bmap_count;		/* # of bitmap blocks. */
+	u32 s_bmap_bits;		/* # of bits in one bitmap blocks */
+	u32 s_last_bmap;
+	struct buffer_head *s_bmap_bh;
+	char *s_prefix;			/* Prefix for volumes and assigns. */
+	int s_prefix_len;		/* Length of prefix. */
+	char s_volume[32];		/* Volume prefix for absolute symlinks. */
+};
+
+#define SF_INTL		0x0001		/* International filesystem. */
+#define SF_BM_VALID	0x0002		/* Bitmap is valid. */
+#define SF_IMMUTABLE	0x0004		/* Protection bits cannot be changed */
+#define SF_QUIET	0x0008		/* chmod errors will be not reported */
+#define SF_SETUID	0x0010		/* Ignore Amiga uid */
+#define SF_SETGID	0x0020		/* Ignore Amiga gid */
+#define SF_SETMODE	0x0040		/* Ignore Amiga protection bits */
+#define SF_MUFS		0x0100		/* Use MUFS uid/gid mapping */
+#define SF_OFS		0x0200		/* Old filesystem */
+#define SF_PREFIX	0x0400		/* Buffer for prefix is allocated */
+#define SF_VERBOSE	0x0800		/* Talk about fs when mounting */
+
+/* short cut to get to the affs specific sb data */
+static inline struct affs_sb_info *AFFS_SB(struct super_block *sb)
+{
+	return sb->s_fs_info;
+}
+
+/* amigaffs.c */
+
+extern int	affs_insert_hash(struct inode *inode, struct buffer_head *bh);
+extern int	affs_remove_hash(struct inode *dir, struct buffer_head *rem_bh);
+extern int	affs_remove_header(struct dentry *dentry);
+extern u32	affs_checksum_block(struct super_block *sb, struct buffer_head *bh);
+extern void	affs_fix_checksum(struct super_block *sb, struct buffer_head *bh);
+extern void	secs_to_datestamp(time_t secs, struct affs_date *ds);
+extern mode_t	prot_to_mode(u32 prot);
+extern void	mode_to_prot(struct inode *inode);
+extern void	affs_error(struct super_block *sb, const char *function, const char *fmt, ...);
+extern void	affs_warning(struct super_block *sb, const char *function, const char *fmt, ...);
+extern int	affs_check_name(const unsigned char *name, int len);
+extern int	affs_copy_name(unsigned char *bstr, struct dentry *dentry);
+
+/* bitmap. c */
+
+extern u32	affs_count_free_blocks(struct super_block *s);
+extern void	affs_free_block(struct super_block *sb, u32 block);
+extern u32	affs_alloc_block(struct inode *inode, u32 goal);
+extern int	affs_init_bitmap(struct super_block *sb, int *flags);
+extern void	affs_free_bitmap(struct super_block *sb);
+
+/* namei.c */
+
+extern int	affs_hash_name(struct super_block *sb, const u8 *name, unsigned int len);
+extern struct dentry *affs_lookup(struct inode *dir, struct dentry *dentry, struct nameidata *);
+extern int	affs_unlink(struct inode *dir, struct dentry *dentry);
+extern int	affs_create(struct inode *dir, struct dentry *dentry, int mode, struct nameidata *);
+extern int	affs_mkdir(struct inode *dir, struct dentry *dentry, int mode);
+extern int	affs_rmdir(struct inode *dir, struct dentry *dentry);
+extern int	affs_link(struct dentry *olddentry, struct inode *dir,
+			  struct dentry *dentry);
+extern int	affs_symlink(struct inode *dir, struct dentry *dentry,
+			     const char *symname);
+extern int	affs_rename(struct inode *old_dir, struct dentry *old_dentry,
+			    struct inode *new_dir, struct dentry *new_dentry);
+
+/* inode.c */
+
+extern unsigned long		 affs_parent_ino(struct inode *dir);
+extern struct inode		*affs_new_inode(struct inode *dir);
+extern int			 affs_notify_change(struct dentry *dentry, struct iattr *attr);
+extern void			 affs_put_inode(struct inode *inode);
+extern void			 affs_delete_inode(struct inode *inode);
+extern void			 affs_clear_inode(struct inode *inode);
+extern void			 affs_read_inode(struct inode *inode);
+extern int			 affs_write_inode(struct inode *inode, int);
+extern int			 affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s32 type);
+
+/* file.c */
+
+void		affs_free_prealloc(struct inode *inode);
+extern void	affs_truncate(struct inode *);
+
+/* dir.c */
+
+extern void   affs_dir_truncate(struct inode *);
+
+/* jump tables */
+
+extern struct inode_operations	 affs_file_inode_operations;
+extern struct inode_operations	 affs_dir_inode_operations;
+extern struct inode_operations   affs_symlink_inode_operations;
+extern struct file_operations	 affs_file_operations;
+extern struct file_operations	 affs_file_operations_ofs;
+extern struct file_operations	 affs_dir_operations;
+extern struct address_space_operations	 affs_symlink_aops;
+extern struct address_space_operations	 affs_aops;
+extern struct address_space_operations	 affs_aops_ofs;
+
+extern struct dentry_operations	 affs_dentry_operations;
+extern struct dentry_operations	 affs_dentry_operations_intl;
+
+static inline void
+affs_set_blocksize(struct super_block *sb, int size)
+{
+	sb_set_blocksize(sb, size);
+}
+static inline struct buffer_head *
+affs_bread(struct super_block *sb, int block)
+{
+	pr_debug("affs_bread: %d\n", block);
+	if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size)
+		return sb_bread(sb, block);
+	return NULL;
+}
+static inline struct buffer_head *
+affs_getblk(struct super_block *sb, int block)
+{
+	pr_debug("affs_getblk: %d\n", block);
+	if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size)
+		return sb_getblk(sb, block);
+	return NULL;
+}
+static inline struct buffer_head *
+affs_getzeroblk(struct super_block *sb, int block)
+{
+	struct buffer_head *bh;
+	pr_debug("affs_getzeroblk: %d\n", block);
+	if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size) {
+		bh = sb_getblk(sb, block);
+		lock_buffer(bh);
+		memset(bh->b_data, 0 , sb->s_blocksize);
+		set_buffer_uptodate(bh);
+		unlock_buffer(bh);
+		return bh;
+	}
+	return NULL;
+}
+static inline struct buffer_head *
+affs_getemptyblk(struct super_block *sb, int block)
+{
+	struct buffer_head *bh;
+	pr_debug("affs_getemptyblk: %d\n", block);
+	if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size) {
+		bh = sb_getblk(sb, block);
+		wait_on_buffer(bh);
+		set_buffer_uptodate(bh);
+		return bh;
+	}
+	return NULL;
+}
+static inline void
+affs_brelse(struct buffer_head *bh)
+{
+	if (bh)
+		pr_debug("affs_brelse: %lld\n", (long long) bh->b_blocknr);
+	brelse(bh);
+}
+
+static inline void
+affs_adjust_checksum(struct buffer_head *bh, u32 val)
+{
+	u32 tmp = be32_to_cpu(((__be32 *)bh->b_data)[5]);
+	((__be32 *)bh->b_data)[5] = cpu_to_be32(tmp - val);
+}
+static inline void
+affs_adjust_bitmapchecksum(struct buffer_head *bh, u32 val)
+{
+	u32 tmp = be32_to_cpu(((__be32 *)bh->b_data)[0]);
+	((__be32 *)bh->b_data)[0] = cpu_to_be32(tmp - val);
+}
+
+static inline void
+affs_lock_link(struct inode *inode)
+{
+	down(&AFFS_I(inode)->i_link_lock);
+}
+static inline void
+affs_unlock_link(struct inode *inode)
+{
+	up(&AFFS_I(inode)->i_link_lock);
+}
+static inline void
+affs_lock_dir(struct inode *inode)
+{
+	down(&AFFS_I(inode)->i_hash_lock);
+}
+static inline void
+affs_unlock_dir(struct inode *inode)
+{
+	up(&AFFS_I(inode)->i_hash_lock);
+}
+static inline void
+affs_lock_ext(struct inode *inode)
+{
+	down(&AFFS_I(inode)->i_ext_lock);
+}
+static inline void
+affs_unlock_ext(struct inode *inode)
+{
+	up(&AFFS_I(inode)->i_ext_lock);
+}
fs/affs/amigaffs.c +1 -8
diff -urN RC10-rc2-bk12-base/fs/affs/amigaffs.c RC10-rc2-bk12-current/fs/affs/amigaffs.c
@@ -8,14 +8,7 @@
  *  Please send bug reports to: [email protected]
  */
 
-#include <stdarg.h>
-#include <linux/stat.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/string.h>
-#include <linux/mm.h>
-#include <linux/amigaffs.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
 
 extern struct timezone sys_tz;
 
fs/affs/bitmap.c +1 -9
diff -urN RC10-rc2-bk12-base/fs/affs/bitmap.c RC10-rc2-bk12-current/fs/affs/bitmap.c
@@ -7,15 +7,7 @@
  *  block allocation, deallocation, calculation of free space.
  */
 
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/stat.h>
-#include <linux/kernel.h>
-#include <linux/slab.h>
-#include <linux/string.h>
-#include <linux/bitops.h>
-#include <linux/amigaffs.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
 
 /* This is, of course, shamelessly stolen from fs/minix */
 
fs/affs/dir.c +1 -11
diff -urN RC10-rc2-bk12-base/fs/affs/dir.c RC10-rc2-bk12-current/fs/affs/dir.c
@@ -13,17 +13,7 @@
  *
  */
 
-#include <asm/uaccess.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/kernel.h>
-#include <linux/affs_fs.h>
-#include <linux/stat.h>
-#include <linux/string.h>
-#include <linux/mm.h>
-#include <linux/amigaffs.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
 
 static int affs_readdir(struct file *, void *, filldir_t);
 
fs/affs/file.c +1 -18
diff -urN RC10-rc2-bk12-base/fs/affs/file.c RC10-rc2-bk12-current/fs/affs/file.c
@@ -12,24 +12,7 @@
  *  affs regular file handling primitives
  */
 
-#include <asm/div64.h>
-#include <asm/uaccess.h>
-#include <asm/system.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/fcntl.h>
-#include <linux/kernel.h>
-#include <linux/errno.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/smp_lock.h>
-#include <linux/dirent.h>
-#include <linux/fs.h>
-#include <linux/amigaffs.h>
-#include <linux/mm.h>
-#include <linux/highmem.h>
-#include <linux/pagemap.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
 
 #if PAGE_SIZE < 4096
 #error PAGE_SIZE must be at least 4096
fs/affs/inode.c +1 -21
diff -urN RC10-rc2-bk12-base/fs/affs/inode.c RC10-rc2-bk12-current/fs/affs/inode.c
@@ -10,26 +10,7 @@
  *  (C) 1991  Linus Torvalds - minix filesystem
  */
 
-#include <asm/div64.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/string.h>
-#include <linux/genhd.h>
-#include <linux/amigaffs.h>
-#include <linux/major.h>
-#include <linux/blkdev.h>
-#include <linux/init.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
-#include <asm/system.h>
-#include <asm/uaccess.h>
-#include <linux/module.h>
+#include "affs.h"
 
 extern struct inode_operations affs_symlink_inode_operations;
 extern struct timezone sys_tz;
@@ -428,4 +409,3 @@
 	affs_unlock_link(inode);
 	goto done;
 }
-MODULE_LICENSE("GPL");
fs/affs/namei.c +1 -14
diff -urN RC10-rc2-bk12-base/fs/affs/namei.c RC10-rc2-bk12-current/fs/affs/namei.c
@@ -8,23 +8,10 @@
  *  (C) 1991  Linus Torvalds - minix filesystem
  */
 
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/kernel.h>
-#include <linux/string.h>
-#include <linux/stat.h>
-#include <linux/fcntl.h>
-#include <linux/amigaffs.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
-#include <asm/uaccess.h>
-
-#include <linux/errno.h>
+#include "affs.h"
 
 typedef int (*toupper_t)(int);
 
-extern struct inode_operations affs_symlink_inode_operations;
-
 static int	 affs_toupper(int ch);
 static int	 affs_hash_dentry(struct dentry *, struct qstr *);
 static int       affs_compare_dentry(struct dentry *, struct qstr *, struct qstr *);
fs/affs/super.c +2 -18
diff -urN RC10-rc2-bk12-base/fs/affs/super.c RC10-rc2-bk12-current/fs/affs/super.c
@@ -11,26 +11,10 @@
  */
 
 #include <linux/module.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/string.h>
-#include <linux/genhd.h>
-#include <linux/amigaffs.h>
-#include <linux/major.h>
-#include <linux/blkdev.h>
 #include <linux/init.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
-#include <linux/vfs.h>
+#include <linux/statfs.h>
 #include <linux/parser.h>
-#include <asm/system.h>
-#include <asm/uaccess.h>
+#include "affs.h"
 
 extern struct timezone sys_tz;
 
fs/affs/symlink.c +1 -8
diff -urN RC10-rc2-bk12-base/fs/affs/symlink.c RC10-rc2-bk12-current/fs/affs/symlink.c
@@ -8,14 +8,7 @@
  *  affs symlink handling code
  */
 
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/stat.h>
-#include <linux/affs_fs.h>
-#include <linux/amigaffs.h>
-#include <linux/pagemap.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
 
 static int affs_symlink_readpage(struct file *file, struct page *page)
 {
include/linux/affs_fs.h +0 -89

94 lines not on this page · raw message

include/linux/affs_fs_i.h +0 -59

59 lines not on this page · raw message

include/linux/affs_fs_sb.h +0 -57

57 lines not on this page · raw message

include/linux/amigaffs.h +0 -103

115 lines not on this page · raw message

#56 lore raw
In reply to a message not in the archive
Quoted 4 lines · Alexandre Oliva

I don't see it as obvious at all. The need for an agreement between two parties on an ABI doesn't imply that one party gets to define it and the other gets to follow it.

Sorry, but that's not how it works.

He who writes the code decides what it is. In this case, if the kernel does a new extension, it's the kernel that gets to decide what it is. Full stop.

If glibc wants to do something new, go wild. The kernel won't care.

And that's really the fundamental issue. The kernel does not care what user land does. The kernel exports functionality, the kernel does _not_ ask user land to help.

That _does_ make it a one-way street. Sorry.

Linus

#176 lore raw
In reply to H. Peter Anvin
Quoted 2 lines · H. Peter Anvin

Most people seem to have suggested include/linux-abi for this; I would personally prefer include/linux-abi/arch for the second.

It seems like there are two ways to adjust the headers. We could move the private headers to a new directory (include/kernel?), or we could move the public headers to a new directory (include/linux-abi?). I am willing to work on some simple and trivial patches to begin doing either one, if we can reach an agreement as to which one is preferrable (and what to call the new directory.)

Quoted 7 lines
Quoted 4 lines

(a) Transfer this stuff into a file of the same name in the new directory. So, for example, the syscall number list from include/asm-i386/unistd.h will be transferred to include/user-i386/unistd.h.

I'm not sure you can do such a 1:1 mapping. In fact, there are cases where you definitely don't want to, because the current structure doesn't make much sense.

I think that the initial trivial patches would probably preserve a 1:1 mapping, just for compatibility. On the other hand, future patches might rearrange things to make more logical sense.

Quoted 13 lines
Quoted 9 lines
     (d) stdint types should be used where possible.

		[include/user-i386/termios.h]
		struct winsize {
			uint16_t ws_row;
			uint16_t ws_col;
			uint16_t ws_xpixel;
			uint16_t ws_ypixel;
		};

Good, except your "struct winsize" is bad; you're stepping on namespace which belongs to userspace. Since we can't use typedefs on struct tags, I suggest:

Quoted 7 lines
struct __kstruct_winsize {
       /* ... */
};

.. and userspace can do:

#define __kstruct_winsize winsize

My initial suggestion would be to leave the types as-is, primarily for the reasons in Linus' earlier email, but also for simplicity and to prevent inadvertent breakage.

Quoted 6 lines
Quoted 3 lines

(3) Remove all #if(n)def __KERNEL__ clauses.

(4) Remove the -D__KERNEL__ from the master kernel Makefile.

Bad! There is code in the kernel which can compile in userspace for testing. This is highly valuable and should be kept.

I would propose that if we decide to move the public headers instead of the internal kernel headers, we autogenerate headers for installation that have a #warning wrapper if __KERNEL__ isn't defined.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a17 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

#57 lore raw
In reply to Linus Torvalds
Followup to:  <[email protected]>
By author:    Linus Torvalds <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 9 lines

If glibc wants to do something new, go wild. The kernel won't care.

And that's really the fundamental issue. The kernel does not care what user land does. The kernel exports functionality, the kernel does _not_ ask user land to help.

That _does_ make it a one-way street. Sorry.

And it SHOULD be a one-way street. A lot of the ugliness in the current stuff comes from the fact that the kernel tries to export libc4/5 internals, instead of the real kernel ABI.

-hpa

#177 lore raw
In reply to Kyle Moffett
Followup to:  <[email protected]>
By author:    Kyle Moffett <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 14 lines · H. Peter Anvin
Quoted 2 lines

Most people seem to have suggested include/linux-abi for this; I would personally prefer include/linux-abi/arch for the second.

It seems like there are two ways to adjust the headers. We could move the private headers to a new directory (include/kernel?), or we could move
the public headers to a new directory (include/linux-abi?). I am willing to
work on some simple and trivial patches to begin doing either one, if we can reach an agreement as to which one is preferrable (and what to call the new directory.)

If we can preserve compatibility, moving the private headers to a separate place makes sense. However, preserving compatibility is not a good thing, because a lot of the brokenness is in how things are exported. Thus, you want libc to be able to have a linux/* wrapper for backwards compatibility.

My first choice would be to put the ABI headers in linux/abi.

Quoted 8 lines
Quoted 8 lines

Good, except your "struct winsize" is bad; you're stepping on namespace which belongs to userspace. Since we can't use typedefs on struct tags, I suggest:

struct __kstruct_winsize {
       /* ... */
};
Quoted 7 lines
Quoted 3 lines

.. and userspace can do:

#define __kstruct_winsize winsize

My initial suggestion would be to leave the types as-is, primarily for the reasons in Linus' earlier email, but also for simplicity and to prevent inadvertent breakage.

True; I hadn't considered the weirdness of the namespace pollution issue. However, the issue with struct tags remain.

Note also that not all ABI issues can be well-described in C headers, and certainly not the way it currently is. Syscall numbers and ioctl APIs are examples there (I do some hideously ugly hacks in klibc to try to auto-derive as much of the ABI as possible; this is part of making klibc very easy to port to new architectures and keeps maintenance in one place.) However, cleaning up the headers is a major step forward.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

(3) Remove all #if(n)def __KERNEL__ clauses.

(4) Remove the -D__KERNEL__ from the master kernel Makefile.

Bad! There is code in the kernel which can compile in userspace for testing. This is highly valuable and should be kept.

I would propose that if we decide to move the public headers instead of the internal kernel headers, we autogenerate headers for installation that have a #warning wrapper if __KERNEL__ isn't defined.

-hpa

#58 lore raw
In reply to Linus Torvalds
Quoted 14 lines · Linus Torvalds
Quoted 4 lines · Alexandre Oliva

I don't see it as obvious at all. The need for an agreement between two parties on an ABI doesn't imply that one party gets to define it and the other gets to follow it.

Sorry, but that's not how it works.

He who writes the code decides what it is. In this case, if the kernel does a new extension, it's the kernel that gets to decide what it is. Full stop.

Quoted 5 lines

If glibc wants to do something new, go wild. The kernel won't care.

And that's really the fundamental issue. The kernel does not care what user land does. The kernel exports functionality, the kernel does _not_ ask user land to help.

except for the user land kernel helpers ;)
via call_usermodehelper()

Quoted 3 lines

That _does_ make it a one-way street. Sorry.

Linus

sorry too,
Herbert

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#59 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

And that's really the fundamental issue. The kernel does not care what user land does. The kernel exports functionality, the kernel does _not_ ask user land to help.

Reminds me of the Klingon programming language:

http://tinyurl.com/4rjco

7 : "Klingon function calls do not have 'parameters' -- they have 'arguments' -- and they ALWAYS WIN THEM."

8 : "What is this talk of 'release'? Klingons do not make software 'releases.' Our software 'escapes' leaving a bloody trail of designers and quality assurance people in its wake."

9 : "Indentation?! - I will show you how to indent when I indent your skull!"

Signature · 2 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
#43 lore raw
In reply to David Woodhouse
Quoted 5 lines · David Woodhouse

I've lost track of the number of times things have broken because of incorrect use of kernel headers from userspace. That's what we're trying to fix -- by putting only the bits which are _supposed_ to be visible into files which userspace sees, where we know they define part of the userspace API and hence we can be extremely careful when editing them.

how about generating the user-visible interfaces and structures from the headers using something like sparse and defined annotations? I.e.:

struct iovec

{
	void __user *iov_base;
	__kernel_size_t iov_len;
} __user_api__;
#44 lore raw

I should have read the whole thread before posting. My apologies, Linus

#15 lore raw
In reply to Matthew Wilcox
Quoted 11 lines · Matthew Wilcox
Quoted 7 lines · David Woodhouse
Quoted 2 lines · David Howells

How about calling the interface headers "kapi*/" instead of "user*/". In case you haven't guessed, "kapi" would be short for "kernel-api".

I don't think that change really makes any difference. The nomenclature really isn't _that_ important.

Indeed. We could also make this transparent to userspace by using a script to copy the user-* headers to /usr/include. Something like this:

And get them promptly out of sync with glibc &c when the kernel changes.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#74 lore raw
In reply to Linus Torvalds
Quoted 5 lines

But in general, I do kind of like the explicit marking. The same way we explicitly mark the functions inside the kernel that we expose to modules, we could try to mark the data structures and values that we expose to user space. That tends to "work".

That gets trickier when it comes to #defines, I think.

Do you really object that much to splitting the header files as I proposed? This should obviate the requirement for such marking (because it's implicit in the location). Getting rid of the #ifdefs would also make the "Thou shalt not use #ifdef" camp happier.

Doing this really shouldn't break the syscall API/ABI at all; barring of course the potential change of location within /usr/include/ of the installed headers, but that is simple to address. All this does is to separate out the userspace available structures and constants.

I can grant that you might not want to use stdint types because not all the toolchains that might be used on the kernel support them, but that's not a reason to throw out the whole proposal.

Personally, I'd prefer us to move to using standard C99 types in lieu of u32 and co at least for the interface to userspace because they are just that:
standard.

However, I can see that there may be objections due to the fact that types may become different logical sizes, and thus cause warnings in printf-format string checking and pointer casting.

Also, the comparison you made to modules isn't exactly valid. We annotate variables and functions that modules are allowed to access, but we never export these to userspace directly; and we don't annotate structures, inline functions, constants and macros into which categories the user-visible stuff falls for use by modules.

David

#100 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

In fact, in many ways I'd prefer to have source-level annotations like "this is exported to user space" over trying to gather things in one place.

I don't think you would; not once you really tried it.

That's what the littering of ifdef __KERNEL__ attempts to do, and there's not really any better way of doing it than that. Whatever form of annotation you came up with, it wouldn't be much different to using __KERNEL__ and performing the 'export' step using unifdef. Changing the sense of it so that we annotate the stuff which _is_ visible rather than the stuff which _isn't_ doesn't really make it fundamentally different either.

If we want to make stuff cleaner in userspace that means spreading the headers much more liberally with such ifdefs or whatever trick we use to replace them. It gets ugly quite fast -- and we already know that the fix for having too many ifdefs is to split stuff up more sensibly. By splitting the visible parts into separate files we actually help to make people _think_ when they're exposing stuff to userspace, and it would help to direct particularly close attention to reviewing structures therein. For example, if the netfilter people tried to add this to a header file which is explicitly for userspace consumption, rather than hidden away in kernel private headers and just _happening_ to be part of the userspace ABI, they'd never get away with it:

struct ipt_rateinfo {
	u_int32_t avg;    /* Average secs between packets * scale */
        u_int32_t burst;  /* Period multiplier for upper limit. */
 
        /* Used internally by the kernel */
        unsigned long prev;
        u_int32_t credit;
        u_int32_t credit_cap, cost;
 
        /* Ugly, ugly fucker. */
        struct ipt_rateinfo *master;
};

Use of an 'unsigned long' which gratuitously changes the structure size between 32-bit userspace and 64-bit kernel, without even any _reason_ for it being there. That kind of thing would be far more obvious if the headers are split, and much less so if we continue as we are.

That and the ugliness of the __KERNEL__ or whatever other annotation scheme we come up with are the main reasons why I think it wants to be separate header files for sharing with userspace, rather than continuing with the mess we have.

Let's go through the kernel headers and identify what actually needs to be exported; yes. But let's not 'annotate' it and post-process the mess of header files to sanitise it; let's just take the parts which need to be shared, put them into different header files and include them from there.

Signature · 1 line
dwmw2
#75 lore raw
In reply to David Howells
Quoted 9 lines · David Howells
Quoted 5 lines

But in general, I do kind of like the explicit marking. The same way we explicitly mark the functions inside the kernel that we expose to modules, we could try to mark the data structures and values that we expose to user space. That tends to "work".

That gets trickier when it comes to #defines, I think.

Do you really object that much to splitting the header files as I proposed?

I object sternuously to your "the header files". If you can't even say _which_ header file, I'm not interested.

If you can say "these X header files have this specific problem Y, and if we move part Z into a common area A, we'd solve it because B", that's a different matter.

See what I'm saying? Whole-sale "move things around because we want to" I'm not interested in. Specific problems, yes.

Quoted 3 lines

Personally, I'd prefer us to move to using standard C99 types in lieu of u32 and co at least for the interface to userspace because they are just that:
standard.

No. I told you why it cannot and MUST NOT be done. Repeat after me:

WE MUST NOT POLLUTE THE NAMESPACE!

There are many cases where including a header file means you want the structure, but it does _not_ mean that you can expose totally unrelated types, even if those types happen to be used as part of the structure. See what I'm saying?

In contrast, __u8/__u16/__xxx are explicitly defined by the relevant standards to be "system name space", and that is _exactly_ why we don't use the standard namespaces.

And the fact is, the rules for _when_ you export _which_ standard types are just too damn hard, and more importantly, they are not something that the kernel headers should even be deciding. They depend on glibc internal things like __USE_BSD etc, and they depend on which standard you compile for, and sometimes even on which _version_ of the standard you compile for.

I will not have that mess in the kernel. Which means that we do _not_ export any of the standard names. Which means that we can't _use_ any of them. End of story.

Linus

#101 lore raw
In reply to David Woodhouse
Quoted 10 lines · David Woodhouse
Quoted 3 lines · Linus Torvalds

In fact, in many ways I'd prefer to have source-level annotations like "this is exported to user space" over trying to gather things in one place.

I don't think you would; not once you really tried it.

That's what the littering of ifdef __KERNEL__ attempts to do, and there's not really any better way of doing it than that.

The fact is, despite this stupid and way-too-long thread, #ifdef __KERNEL__ has worked for over a decade, and works damn well, everything considered.

Remember the second-system-syndrome? It comes from people wanting to fix problems in the current implementation, without thinking about all the things that it does _well_ (because the things that work are not on their radar - they just work). And no, __KERNEL__ may not be pretty, but it's damn simple to fix up, and one thing it does really well is allow flexibility in a way that forcing structure does not.

I know people like "structure". But it's _waayyy_ overrated. The reason we use C instead of some other programming environment is not that C is the most highly structured language around, but it's the most _flexible_ one, largely because it says "let's give people rope". It allows structure, but if you want to cast things around and use gotos, it says "sure, master, you're the boss".

Same thing here. The __KERNEL__ approach says "whatever you want, boss". It doesn't get in the way. Maybe it doesn't actively _help_ you either, but you never have to fight any structure it imposes on you.

Also, there _are_ better ways of annotating it than __KERNEL__. In particular, if we had a "user annotation", I could make sparse sit up and take notice, and _complain_ when you use a non-specific-sized type. That's not just theory - Al Viro was talking about that at some point.

Linus

#76 lore raw
In reply to Linus Torvalds
Quoted 4 lines

No. I told you why it cannot and MUST NOT be done. Repeat after me:

WE MUST NOT POLLUTE THE NAMESPACE!

Sorry, I misremembered that bit in your earlier email, but having reread it I think I have to concur with it. I don't like it very much, but I don't think there's a good way around it now.

David

#102 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

The fact is, despite this stupid and way-too-long thread, #ifdef __KERNEL__ has worked for over a decade, and works damn well, everything considered.

I don't agree with that. The current setup is a complete PITA and that's why people are clamouring for us to clean the crap up.

But if you feel this strongly about it, then let's go ahead with the existing mess. We can add more ifdef __KERNEL__ and add rules to the makefiles to export it for userspace, generating something like the existing glibc-kernheaders setup. And of course we can look at the better alternative 'annotations' instead of using #ifdef __KERNEL__.

I think we'll all hate it, and I don't think it's going to really fix the problem. But we can always reconsider the position later.

Quoted 3 lines

Same thing here. The __KERNEL__ approach says "whatever you want, boss". It doesn't get in the way. Maybe it doesn't actively _help_ you either, but you never have to fight any structure it imposes on you.

Having to think before adding something that's user visible is a _benefit_ not a disadvantage.

Signature · 1 line
dwmw2
#128 lore raw
In reply to Linus Torvalds
Quoted 4 lines

The fact is, despite this stupid and way-too-long thread, #ifdef __KERNEL__ has worked for over a decade, and works damn well, everything considered.

It's a pain, which is why people want to do something about it.

Quoted 6 lines

Remember the second-system-syndrome? It comes from people wanting to fix problems in the current implementation, without thinking about all the things that it does _well_ (because the things that work are not on their radar - they just work). And no, __KERNEL__ may not be pretty, but it's damn simple to fix up, and one thing it does really well is allow flexibility in a way that forcing structure does not.

And allows abuse and doesn't penalise laziness; both of which do happen in the kernel header files.

Quoted 5 lines

I know people like "structure". But it's _waayyy_ overrated.

The reason we use C instead of some other programming environment is not that C is the most highly structured language around, but it's the most _flexible_ one, largely because it says "let's give people rope".

We use C & ASM for the kernel because you and others won't allow anything else. Not that I'm necessarily endorsing the use of other languages - I like C and ASM; I just look at stuff in the kernel occasionally and know that other languages handle it better and more efficiently. That is, however, besides the point.

Quoted 3 lines

Same thing here. The __KERNEL__ approach says "whatever you want, boss". It doesn't get in the way. Maybe it doesn't actively _help_ you either, but you never have to fight any structure it imposes on you.

It can get in the way accidentally.

Quoted 4 lines

Also, there _are_ better ways of annotating it than __KERNEL__. In particular, if we had a "user annotation", I could make sparse sit up and take notice, and _complain_ when you use a non-specific-sized type. That's not just theory - Al Viro was talking about that at some point.

If you did user annotations you'd have to solve the problem of applying it to #defines and still allowing the constants to be used in assembly. Obviously this is not impossible. This is trivial with __KERNEL__ or separation into other files.

David

#103 lore raw
In reply to David Woodhouse
Quoted 7 lines · David Woodhouse
Quoted 4 lines

Same thing here. The __KERNEL__ approach says "whatever you want, boss". It doesn't get in the way. Maybe it doesn't actively _help_ you either, but you never have to fight any structure it imposes on you.

Having to think before adding something that's user visible is a _benefit_ not a disadvantage.

I've said this at least three times: if you can point to a _specific_ thing you want to move, go wild. I think the big waste in this discussion has been that there have _not_ been specific suggestions, just total sound-bites like "wouldn't it be great to move things to 'include/kapi'".

If you have a specific thing in mind, say instead something like

"Wouldn't it be great if we moved all the tty layer IOCTL numbers
into 'tty-ioctl-nr.h', and made the old header file just include that header file, so that new libc users can get them from just that header? And btw, here's the patch."

then I might listen. Notice how the only really constructive thing to come out of this flame-fest has been a patch by Al that looked perfectly reasonable, but that got totally drowned out by the arguing?

Note that even _if_ you have a specific thing in mind, I want to see that somebody would say "yes, we'd use that organization". I would not be surprised at all if glibc people said that they can't really use any re-organization anyway, since they need to support old kernel setups too.

See? Changes that aren't specific enough, or don't actually help things is what I'm against.

Linus

#129 lore raw
In reply to David Howells
Quoted 5 lines · David Howells

If you did user annotations you'd have to solve the problem of applying it to #defines and still allowing the constants to be used in assembly. Obviously this is not impossible. This is trivial with __KERNEL__ or separation into other files.

Annotations don't have to be per-entry. It can be something as simple as "this particular file is exposed to user space", or "things from here to here are exposed to user space".

And while automation is good, sometimes automation isn't even worth it. I suspect it might be good just to _comment_ the structures we expose to user space, even if it's never used for anything but as a note to tell people not to change it.

Linus

#104 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

I've said this at least three times: if you can point to a _specific_ thing you want to move, go wild. I think the big waste in this discussion has been that there have _not_ been specific suggestions, just total sound-bites like "wouldn't it be great to move things to 'include/kapi'".

I've already done some -- see the contents of include/mtd/ for example, which contain only the parts which userspace should see, while include/linux/mtd/ is now entirely kernel-private stuff.

We were trying to reach a consensus on where new header files can be put and indeed whether they should be split at all. We can all disappear for a week or two and come back with a completely sanitised set of header files with the userspace stuff in a separate directory and _then_ and be told "no, we don't like that; do it differently".

A bunch of us have even made a start on such things to prove the concept. Presenting it as a fait accompli just to have it rejected isn't much fun though -- the idea was to get some form of agreement in principle beforehand, then collaborate on actually doing it.

Quoted 4 lines

Note that even _if_ you have a specific thing in mind, I want to see that somebody would say "yes, we'd use that organization". I would not be surprised at all if glibc people said that they can't really use any re-organization anyway, since they need to support old kernel setups too.

The idea in the proposal which David posted, which seemed perfectly specific enough to me, was to move all the user-visible parts to separate header files in a separate directory. Those same header files would also be included by the kernel-private headers. This would be done in such a way that it's a drop-in replacement for the existing sanitised glibc-kernheaders package. Of _course_ it needs to be usable without pain by the glibc people.

We can just go ahead and _do_ that if necessary, but it seemed like a sane plan to agree on how to do it beforehand.

Signature · 1 line
dwmw2
#127 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

then I might listen. Notice how the only really constructive thing to come out of this flame-fest has been a patch by Al that looked perfectly reasonable, but that got totally drowned out by the arguing?

Frankly, I suspect that amount of effort that went into postings made in that thread is approaching what it would take to clean the damn thing up for good. I wasn't kidding about 10 minutes work to do that patch...

#105 lore raw
In reply to David Woodhouse
Quoted 4 lines · David Woodhouse

The idea in the proposal which David posted, which seemed perfectly specific enough to me, was to move all the user-visible parts to separate header files in a separate directory.

You call _that_ specific?

Hell no. You need to do it without breaking existing uses, as noted earlier, and it's not specific at all. "all user visible parts" is a big undertaking, if you can't make it smaller than that, then forget about it.

Basic rule in kernel engineering: you don't just rewrite the world. You do it in incremental independent steps.

Any mtd-specific rewrite is obviously a go.

Linus

#106 lore raw
Quoted 10 lines · Linus Torvalds

You call _that_ specific?

Hell no. You need to do it without breaking existing uses, as noted earlier, and it's not specific at all. "all user visible parts" is a big undertaking, if you can't make it smaller than that, then forget about it.

Basic rule in kernel engineering: you don't just rewrite the world. You do it in incremental independent steps.

Any mtd-specific rewrite is obviously a go.

Facts:
- current __KERNEL__ stuff is crap; just check any distro what they're using - either some kind of userland headers or a patched version of kernel headers
- "You don't use kernel headers in userland!!"
small print: except this, this, and that dir, which are userland friendly That's quite schizophrenic.

Specific problems:
- glibc has a copy of eg. networking definitions (mostly lots of numbers for different protocols); if userland wants to use something that's not in glibc already, it has to include linux' headers; which more often than not causes conflicts with glibc
- the above problem is present in allmost all headers that have a glibc and linux version
- glibc uses linux headers for getting ABI stuff in... dunno... four? five cases? And *everything* else ends up getting duplicated.

People in this thread are trying to force you to agree to a specific location where stuff like the above mentioned mtd can go to and to start accepting patches (afaik there were a number of patches trying to introduce that userland dir - all of them ignored). That's (mostly) all.

Signature · 5 lines
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
#77 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

I object sternuously to your "the header files". If you can't even say _which_ header file, I'm not interested.

How about this formulation then:

- move anything that is not protected by #ifdef __KERNEL__ to the ukabi header tree, adding an include in the beginning of the original header that includes the ukabi header. Since ukabi stuff can't depend on kernel-internal stuff, having this include as the first thing in the existing header should work.

- figure out what should have been protected by #ifdef __KERNEL__ but wasn't, and move it back

- wait for bug reports from the world and deal with them

- repeat until you feel you got it right

Quoted 2 lines

See what I'm saying? Whole-sale "move things around because we want to" I'm not interested in. Specific problems, yes.

The specific problem we're trying to address is the creation of header files that define the ABI between userland and kernel, that both kernel and userland can use.

Quoted 7 lines
Quoted 3 lines

Personally, I'd prefer us to move to using standard C99 types in lieu of u32 and co at least for the interface to userspace because they are just that:
standard.

No. I told you why it cannot and MUST NOT be done. Repeat after me:

WE MUST NOT POLLUTE THE NAMESPACE!

David, Linus is right. Using standard types is not an option because they're not built in type names; you have to include a header to bring them in, and ideally you shouldn't gratuitously get the names defined just because you include some unrelated header, although the C Standard grants standard headers freedom to include other headers or something along these lines; I don't recall the exact passage of the Standard that says so, but we should strive to not pollute the user namespace anyway.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#107 lore raw
In reply to Mariusz Mazur
Quoted 3 lines · Mariusz Mazur

Facts:
- current __KERNEL__ stuff is crap;

Fact: you're wrong. I _constantly_ get emails when something breaks. It doesn't happen all that often, but it happens.

IOW, I _know_ from personal experience that people depend on it. No ifs, buts and maybes about it.

Quoted 4 lines

People in this thread are trying to force you to agree to a specific location where stuff like the above mentioned mtd can go to and to start accepting patches (afaik there were a number of patches trying to introduce that userland dir - all of them ignored). That's (mostly) all.

No. People in this thread have almost uniformly ignored my arguments.

If it's MTD-specific, nobody _cares_ where it goes. Somebody take a fricking flying leap of faith here.

If that's all that people want, I hereby proclaim that

include/asm-xxx/user/xxxx.h
include/user/xxxx.h

is reserved for user-visible stuff. And now you can send me small and localized patches that fix a _particular_ gripe.

But claiming that __KERNEL__ doesn't work is clearly a bunch of crapola. As is the notion that you can somehow do it all. We do it in small pieces.

Happy now?

Linus

#78 lore raw
In reply to Alexandre Oliva
Quoted 4 lines · Alexandre Oliva

- move anything that is not protected by #ifdef __KERNEL__ to the ukabi header tree, adding an include in the beginning of the original header that includes the ukabi header.

No. I want stuff that goes into the ABI tree to be clearly _defined_ to be user-visible. Not a "let's move it all there and then prune it".

Leave anything questionable in the current location. And never EVER move anything that is kernel-internal to a new "clean" tree, because that would be totally pointless. At that point, people would have to edit the "clean" tree even for kernel internal stuff. No go.

Also, "ukabi" just isn't going to fly as a name. It's also not as simple as you seem to think, since a lot of these ABI things are architecture- dependent, which apparently all you guys have totally ignored.

I've suggested "include/user/" and "include/asm-xxx/user", which handles architecture-specific parts too. I'm ok with doing it the other way around, ie "include/user/" and "include/user/arch-xxxx".

And "user" might be "user-abi" or something like that, but it sure isn't going to be some unreadable contraction.

And I _still_ want to see these patches only for things where somebody can validly argue that
(a) it can't break anything (ie the old location still includes the new one, exactly the same way)
(b) there are people who will actually take _advantage_ of that particular file (ie "just because I think so" doesn't fly).

Moving files around is just too disruptive to be done without damn good reason.

Linus

#60 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alexandre Oliva

I don't see it as obvious at all. The need for an agreement between two parties on an ABI doesn't imply that one party gets to define it and the other gets to follow it.

Sorry, but that's not how it works.

Then maybe this is the fundamental problem. As long as the kernel doesn't recognize that an ABI is a contract, rather than an imposition, kernel developers won't care.

But guess what? Kernel developers *do* care about maintaining the userland ABI. They don't just go wild changing data types, structures, renumbering system calls and all that sort of nasty stuff because there's a contract with userland that they don't want to break. Because breaking this contract would be a stupid thing, that would require all affected userland to be modified to cope with the change, even if that's just a recompile using the new kernel headers... erhm... using the result of applying the kernel ABI change in the copy/extract/whatever of kernel headers that userland is entitled to use.

As long as we recognize that the notion that the kernel<->userland ABI is a separate entity, that enables kernel and userland to exchange information, rather than something that the kernel defines and userland must follow, we give strength to the argument that userland is not a derived work from the kernel. If this argument proves to be false, then only works released under the GNU GPL or a compatible license will be allowed to run atop of Linux, because libc would be a derived work, its license would become full GPL, and any program linked with it could only be distributed under the full GPL. Oh, and just in case, IANAL :-)

Again, I wouldn't mind that at all, but I'm sure others feel otherwise. Besides, as I wrote before, if this is the intent, I'd rather have it exposed in clear, rather than seeing people being misled to believe it's fair game to run non-GPLed software on Linux-based operating systems.

Quoted 1 line

He who writes the code decides what it is.

I've been in projects in which I wrote ABI headers before the corresponding kernel port had even started. Sometimes, quite unfortunately, the kernel engineer went ahead and wrote headers with slightly different assumptions, and then we had to get together to define what the ABI was going to be.

In fact, the way you seem to want things to be, you want both kernel and userland to write the code, and somehow hope they magically match each other. That's not how things work. The way things work is that either you get agreement first, or each end goes on its way and then at some point they figure out where they diverged and fix things up, or one end goes first and the other takes the definitions and uses them. It's in no way a kernel->userland one-way street. It's a language (for lack of a better word) that kernel and userland choose to exchange information, not one that the kernel invents and then requires others interested in talking to it to learn (you know, that ``Do you speak English/my language?´´ phrase so common in Hollywood movies in which an American character needs to talk to someone abroad :-)

Sometimes you'll even find formal ABI specifications, written down before either userland or kernel start being developed. Claiming that the kernel gets to set the ABI as it wishes, without worrying about userland because it never calls userland is a very narrow view, and in some cases it's even a lie. Think signal handling and process start-up, for example: if you don't follow the userland ABI to call the function pointer given to the signal function, or if you don't follow the userland ABI to start the program's entry point, it's not userland that is broken, it's the kernel.

Quoted 2 lines

In this case, if the kernel
does a new extension, it's the kernel that gets to decide what it is.

As in, say, NPTL? As if a number of userland considerations hadn't driven most of the design decisions behind the then-defined ABI extensions?

Quoted 1 line

If glibc wants to do something new, go wild. The kernel won't care.

This is not about glibc. That's not the only userland component. There are several different alternatives one can use.

And then, the kernel will care, because it has to. A kernel without userland is nothing. A kernel that kept introducing incompatible ABI changes would die a very quick death. A kernel has to be very careful about complying with the ABI it shares with userland, and you know that, because you do care about it.

Quoted 3 lines

And that's really the fundamental issue. The kernel does not care what user land does. The kernel exports functionality, the kernel does _not_ ask user land to help.

I'm not so much talking about the hundreds of syscalls, the syscall conventions and the number and type of arguments given to each syscall. This is not even well defined in kernel headers. All that's exported regarding syscalls is a mapping from syscall names to numbers.

What I'm talking about is the data structures. Sure, the kernel can define these however it likes. So can userland. The point is that, if they don't match, when you get userland and kernel together, in the hopes of getting something useful done, it doesn't work.

Both ends can come up with extensions, and do things the old way until the other end picks up the extension. I'll give you that many extensions make to the kernel before say glibc, but that's mostly just like, when we create a new toolchain port, assembler and linker are generally implemented before the compiler. I guess it just makes more sense. It wastes less effort to make sure you got something right in a lower layer before you build something atop of it. It's unlikely that a change would make to glibc if it used a feature that isn't available in the kernel, just because most often features undergo (minor or sometimes major) changes when they're integrated, and it's easier to fix things up if you haven't committed to the way that didn't make it there yet.

Quoted 1 line

That _does_ make it a one-way street. Sorry.

Good, we're getting somewhere. This is surely better than a zero-way street, i.e., one in which headers that specify the ABI don't flow in any direction. If we're agreeing to make it one way, let's make it
so: have headers that are maintained as part of the kernel, and that are imposed, if you like, on userland. Saying it's a one-way street but denying others from taking what comes their way through this one-way street hasn't been very effective :-)

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#79 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Alexandre Oliva

- move anything that is not protected by #ifdef __KERNEL__ to the ukabi header tree, adding an include in the beginning of the original header that includes the ukabi header.

No. I want stuff that goes into the ABI tree to be clearly _defined_ to be user-visible. Not a "let's move it all there and then prune it".

Right. Which brings us to my second bullet. I *knew* I should have made it the first :-/

Quoted 3 lines

Also, "ukabi" just isn't going to fly as a name. It's also not as simple as you seem to think, since a lot of these ABI things are architecture- dependent, which apparently all you guys have totally ignored.

Looks like you missed the post that started this thread. It *did* mention machine-independent and machine-specific user headers.

Quoted 3 lines

I've suggested "include/user/" and "include/asm-xxx/user", which handles architecture-specific parts too. I'm ok with doing it the other way around, ie "include/user/" and "include/user/arch-xxxx".

As I pointed out, user' is a very bad name. As you said yourself, we're talking about the *kernel* ABI. So what's user' supposed to mean? Was I so successful in my arguments that you now see it as the userland ABI? :-)

Quoted 2 lines

(a) it can't break anything (ie the old location still includes the new one, exactly the same way)

You mean it can't break anything in a kernel build, or it can't break anything except for userland apps that abused kernel headers and used to get away with that?

Quoted 2 lines

(b) there are people who will actually take _advantage_ of that particular file (ie "just because I think so" doesn't fly).

People who currently get to maintain duplicates of these header contents will take immediate advantage of these changes, since they will no longer have to maintain the duplicates. This is a major step forward in terms of maintainability for everybody else, at no expense to the kernel, that now gets to explicitly (as opposed to implicitly) document what is the ABI it's willing to comply with. It's a win for both ends.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#80 lore raw
Quoted 8 lines · Alexandre Oliva
Quoted 3 lines

I've suggested "include/user/" and "include/asm-xxx/user", which handles architecture-specific parts too. I'm ok with doing it the other way around, ie "include/user/" and "include/user/arch-xxxx".

As I pointed out, user' is a very bad name. As you said yourself, we're talking about the *kernel* ABI. So what's user' supposed to mean? Was I so successful in my arguments that you now see it as the userland ABI? :-)

I got carried away with joking and failed to repeat why I consider it a bad name (assuming that, since you missed the beginning of the thread, you probably missed the first reply I posted to the message that started it): since we're going to install these headers in /usr/include (where headers for userland live), /usr/include/user is quite misleading. /usr/include/kernel would be far more appropriate for this purpose.

Sure, we could take headers from linux-*/include/user and install them in /usr/include/kernel, but then includes in there that reference other headers in user/ or in asm-<arch>/ will cease to work.

So we should come up with a name that makes sense for both users of these headers, which is why I suggested ukabi. linux/abi and asm-<mach>/abi work just as well, and then we can soft-link `abi -> .' in /usr/include/{linux,asm-<mach>} if needed. Ideally, we wouldn't have to.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#61 lore raw
In reply to Alexandre Oliva
Quoted 4 lines · Alexandre Oliva

Then maybe this is the fundamental problem. As long as the kernel doesn't recognize that an ABI is a contract, rather than an imposition, kernel developers won't care.

That's a silly analogy. Worse, it's a very flawed analogy.

If you want to use a legal analogy, the ABI is not a contract, it's a public _license_.

Why? A contract is something that you cannot change unilaterally. Clearly the kernel _can_ and will change the ABI without asking every existing program for permission.

In a license, you can always just _expand_ the license eventually. Maybe you originally licensed for "fly-fishing for trout", and later you can expand that license to say "you can also catch crawfish" without impacting your existing licensees.

And exactly as with an ABI, the only thing you can't do is _remove_ rights without getting into trouble.

So get your analogies straight. The kernel ABI is _not_ a contract.

Linus

#82 lore raw
In reply to Alexandre Oliva
Quoted 6 lines · Alexandre Oliva
Quoted 2 lines

(a) it can't break anything (ie the old location still includes the new one, exactly the same way)

You mean it can't break anything in a kernel build, or it can't break anything except for userland apps that abused kernel headers and used to get away with that?

It can't break userland either.

Quoted 6 lines
Quoted 2 lines

(b) there are people who will actually take _advantage_ of that particular file (ie "just because I think so" doesn't fly).

People who currently get to maintain duplicates of these header contents will take immediate advantage of these changes, since they will no longer have to maintain the duplicates.

Wrong. They'll _still_ have to maintain duplicates, since they can't rely ont he end-user to have a recent enough kernel.

It's just that they can hopefully start _copying_ their dupliates more easily. But if you think the duplication goes away, then I don't want you to send me patches, because you don't understand the issues.

At RH you may see only the case where people do a whole-system upgrade. Please realize that that is just _one_ schenario, and you should not assume that it's the only valid one.

Linus

#108 lore raw
In reply to Linus Torvalds

On Tue, Nov 30, 2004 at 12:47:50PM -0800, Linus Torvalds wrote:
If that's all that people want, I hereby proclaim that

Quoted 6 lines

include/asm-xxx/user/xxxx.h
include/user/xxxx.h

is reserved for user-visible stuff. And now you can send me small and localized patches that fix a _particular_ gripe.

Please use:
include/$arch/user-asm/xxxx.h
include/user/xxxx.h

This allows us to

1) To include file foo.h we still distingush between user versus user-asm:
#include <user/foo.h> (include/user/foo.h)
#include <user-asm/foo.h> (include/$arch/user-asm/foo.h)

2) No symlinks needed - just proper options to gcc

Sam

This should do it for the top-level makefile:

===== Makefile 1.549 vs edited =====

Makefile +4 -0
@@ -345,6 +345,10 @@
 # Needed to be compatible with the O= option
 LINUXINCLUDE    := -Iinclude \
                    $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include)
+		   
+# Extend include path with user dir
+LINUXINCLUDE	+= -Iinclude/$(ARCH) \
+		   $(if $(KBUILD_SRC), -I$(srctree)/include/$(ARCH))
 
 CPPFLAGS        := -D__KERNEL__ $(LINUXINCLUDE)
 
#83 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Alexandre Oliva
Quoted 2 lines

(a) it can't break anything (ie the old location still includes the new one, exactly the same way)

You mean it can't break anything in a kernel build, or it can't break anything except for userland apps that abused kernel headers and used to get away with that?

It can't break userland either.

That depends on your definition of 'break'. It should prevent abuse.

To pick a specific example, since you like them: where userland programs are including atomic.h, and hence writing programs which don't compile on some architectures, and which compile on others but silently give non-atomic results, it's perfectly acceptable and indeed advisable to prevent compilation across the board.

Some people might call that breakage; I don't.

Signature · 1 line
dwmw2
#109 lore raw
In reply to Sam Ravnborg
Quoted 6 lines · Sam Ravnborg

On Tue, Nov 30, 2004 at 12:47:50PM -0800, Linus Torvalds wrote:
If that's all that people want, I hereby proclaim that

Quoted 3 lines

include/asm-xxx/user/xxxx.h
include/user/xxxx.h

Quoted 6 lines
Quoted 2 lines

is reserved for user-visible stuff. And now you can send me small and localized patches that fix a _particular_ gripe.

Please use:
include/$arch/user-asm/xxxx.h
include/user/xxxx.h

Wrong. These dirs must be linked to /usr/include so they must have more meaningfull names.

Signature · 5 lines
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
#118 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · David Woodhouse

The idea in the proposal which David posted, which seemed perfectly specific enough to me, was to move all the user-visible parts to separate header files in a separate directory.

Quoted 5 lines

You call _that_ specific?

Hell no. You need to do it without breaking existing uses, as noted earlier, and it's not specific at all. "all user visible parts" is a big undertaking, if you can't make it smaller than that, then forget about it.

So we follow dhowell's plan with the following additions:

a) during the transition, include/linux/foo.h includes include/user/foo.h if it exists and similarly for asm-*
b) when include/user is deemed sufficiently populated, a flag day is declared and links from /usr/include are switched to them
c) #define __KERNEL__ is dropped from the compile
d) extraneous include/user/* includes are removed from include/linux (though they'll often make perfect sense)

Step b) may be a long time in coming, but for interfaces where kernel headers are frequently abused, we can start feeling benefits.

Quoted 2 lines

Basic rule in kernel engineering: you don't just rewrite the world. You do it in incremental independent steps.

Even so, it's sometimes important for everyone to agree on a destination before they set out..

--
Mathematics is the supreme nostalgia of our time.

#84 lore raw
In reply to David Woodhouse
Quoted 2 lines · David Woodhouse

That depends on your definition of 'break'. It should prevent abuse.

Not really.

It should prevent _future_ abuse.

The notion of "preventing existing xxx" is insane. You can't "prevent" something that already happened unless you've come up with some new interesting theory of causality.

Quoted 7 lines

To pick a specific example, since you like them: where userland programs are including atomic.h, and hence writing programs which don't compile on some architectures, and which compile on others but silently give non-atomic results, it's perfectly acceptable and indeed advisable to prevent compilation across the board.

Some people might call that breakage; I don't.

I do. The thing is, the people who _notice_ the breakage are often the people who don't know what the hell to do about it.

The way to prevent _future_ abuse is by adding something like

	#ifndef __KERNEL__
	#warning "This really doesn't work"
	#endif

which does that, and has the advantage of not breaking anything at all.

In other words: if you want to move things around just to break things, THEN THAT IS INCREDIBLY STUPID. We don't do things to screw our users over.

Feel free to send a patch.

Linus

#119 lore raw
In reply to Matt Mackall
Quoted 4 lines

a) during the transition, include/linux/foo.h includes include/user/foo.h if it exists and similarly for asm-*
b) when include/user is deemed sufficiently populated, a flag day is declared and links from /usr/include are switched to them

there are no such links, only copies (more or less modified)

#97 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 3 lines

You mean it can't break anything in a kernel build, or it can't break anything except for userland apps that abused kernel headers and used to get away with that?

It can't break userland either.

Userland? And who's using vanilla kernel headers in userland? Duplicates maintainers can always fix stuff independent of kernel (and I do).

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

(b) there are people who will actually take _advantage_ of that particular file (ie "just because I think so" doesn't fly).

People who currently get to maintain duplicates of these header contents will take immediate advantage of these changes, since they will no longer have to maintain the duplicates.

Wrong. They'll _still_ have to maintain duplicates, since they can't rely ont he end-user to have a recent enough kernel.

The end user uses a distro that does these things for him. If not, he needs to know his way around with finding working kernel headers, otherwise he won't be able to build anything (he surely isn't able now - which btw you're trying really hard to ignore).
I don't get your point.

Quoted 3 lines

It's just that they can hopefully start _copying_ their dupliates more easily. But if you think the duplication goes away, then I don't want you to send me patches, because you don't understand the issues.

Yes. The more stuff gets separated, the less work duplicates maintainers have. Once all userland stuff gets extracted, those maintainers won't have anything else to do, maybe except releasing their final version and using it for backward compatibility. And that's what we're aiming at.

Signature · 5 lines
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
#123 lore raw
In reply to Matt Mackall
Quoted 2 lines · Matt Mackall

So we follow dhowell's plan with the following additions:

No.

We do _not_ move stuff over that is questionable.

I thought that was clear by now. The rules are:
- we only move things that _have_ to move
- we don't break existing programs, and no "but they are broken already" is not an excuse.
- we only move things where that _particular_ move can be shown to be beneficial.

No whole-sale moves. No "let's break things that I think are broken". No "let's change things because we can".

Well-defined moves. Both in content _and_ in reason.

Linus

#62 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Alexandre Oliva

Then maybe this is the fundamental problem. As long as the kernel doesn't recognize that an ABI is a contract, rather than an imposition, kernel developers won't care.

That's a silly analogy. Worse, it's a very flawed analogy.

If you want to use a legal analogy, the ABI is not a contract, it's a public _license_.

I didn't mean to use a legal analogy. I meant contract in the software engineering sense. Sorry if that wasn't clear.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#110 lore raw
In reply to Mariusz Mazur
Quoted 10 lines · Mariusz Mazur
Quoted 9 lines · Sam Ravnborg

On Tue, Nov 30, 2004 at 12:47:50PM -0800, Linus Torvalds wrote:
If that's all that people want, I hereby proclaim that

Quoted 5 lines

include/asm-xxx/user/xxxx.h
include/user/xxxx.h

is reserved for user-visible stuff. And now you can send me small and localized patches that fix a _particular_ gripe.

Quoted 6 lines
Quoted 3 lines

Please use:
include/$arch/user-asm/xxxx.h
include/user/xxxx.h

Wrong. These dirs must be linked to /usr/include so they must have more meaningfull names.

WTF? I've got a dozen kernel trees hanging around, which one (and WTF any, while we are at it) should be "linked to"?

#99 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Wrong. They'll _still_ have to maintain duplicates, since they can't rely ont he end-user to have a recent enough kernel.

No. They just won't have the headers for the newer kernel installed on their system. They stick with the kernel ABI headers provided by a baseline kernel, and that's probably from an older kernel tarball.

Quoted 2 lines

It's just that they can hopefully start _copying_ their dupliates more easily.

It's still the headers from the kernel, copied or not. From a different, baseline version, maybe.

Quoted 1 line

At RH you may see only the case where people do a whole-system upgrade.

I personally don't care about the case you perceive as the Red Hat case, because I work on embedded systems development, in a different organization within Red Hat than the people who develop Red Hat Enterprise Linux and Fedora. So yes, in the case that actually benefits me, it's even more of a whole-system thing, since for our group it's *the* kernel, not *a* random kernel version. It's the kernel we port and ship to the customer, and the corresponding kernel<->userland ABI.

That said, I do understand the issues of kernel baseline ABI settings, or at least part of them. I know you don't want to assume the presence of a certain system call just because you're running a kernel that implements it. In fact, I know you don't want to assume the presence of a system call just because it's defined in linux/unistd.h. As long as the userland software keeps this distinction in mind, I don't see why taking the latest kernel<->userland ABI headers, that describe the newest system calls and any corresponding data structures, could hurt anything. In fact, if you don't have the corresponding system call numbers and data structures, you can't possibly take advantage of the new syscalls even if they happen to be available, unless you duplicate the syscall numbers and data structure definitions in your own code, which, from a software engineering standpoint, is bad.

Of course, badly written software can assume the syscalls are present just because their numbers are given, and then people who rely on such bad software lose, but such is the nature of bad software.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#120 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines

b) when include/user is deemed sufficiently populated, a flag day is declared and links from /usr/include are switched to them

there are no such links, only copies (more or less modified)

This may be somewhat heretical, but someone has to ask...

Once include/user/foo.h is sufficiently clean and sufficiently complete, is there any reason to not allow such links?

Chris

#63 lore raw
In reply to Alexandre Oliva
Quoted 8 lines · Alexandre Oliva
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Alexandre Oliva

Then maybe this is the fundamental problem. As long as the kernel doesn't recognize that an ABI is a contract, rather than an imposition, kernel developers won't care.

Quoted 7 lines
Quoted 4 lines

That's a silly analogy. Worse, it's a very flawed analogy.

If you want to use a legal analogy, the ABI is not a contract, it's a public _license_.

I didn't mean to use a legal analogy. I meant contract in the software engineering sense. Sorry if that wasn't clear.

Then your definition of a "contract" is flawed or your world-view has nothing to do with reality.

An ABI is not a contract. It's somebody exposing a certain set of features. It does _not_ mean that the party that gets exposed has any say wrt the features. At most he can hope that the exported set doesn't shrink or change semantically, but let's face it, even that does happen.

The Linux kernel has been very very good about this exposure, in the sense that we try very very hard not to break it. I think that's a good thing. But that in no way makes anything a "contract".

It's the same thing with CPU vendors. You as a programmer, didn't get to dictate what the CPU vendor implemented. You take it or leave it, and at most you can make suggestions about things that might make the CPU vendor sell more chips. The ISA is not a "contract" - it's just an interface that was handed down to you. You didn't get to negotiate, you got to say "yes" or "no".

Linus

#98 lore raw
In reply to Mariusz Mazur
Quoted 3 lines · Mariusz Mazur

Userland? And who's using vanilla kernel headers in userland? Duplicates maintainers can always fix stuff independent of kernel (and I do).

Dammit, how hard is it for people to admit that they aren't the only ones around?

There are still people who use ancient setups, and upgrade the kernel. They won't complain to _you_ or to redhat when their setup breaks, because they aren't _using_ your setup. They complain to me.

This is what statisticians call "self-selection". It means that you guys only see the part of the world that uses your stuff, so you think that there is nothing else.

Quoted 1 line

I don't get your point.

Very obviously.

Linus

#111 lore raw
In reply to Al Viro
Quoted 2 lines · Al Viro

WTF? I've got a dozen kernel trees hanging around, which one (and WTF any, while we are at it) should be "linked to"?

Whichever you chose to install in your /usr/include, and use as the kernel ABI definition as far as userland is concerned.

I don't think `make install' should touch /usr/include at all. It should be a separate step, such that one can build a kernel abi headers package out of the kernel source tree, ideally without even having to configure it first, and use that as the kernel ABI definition. You should only have to do this again *if* the ABI changes (ideally no removals, only additions) *and* you want to take advantage of them, *and* you're willing to rebuild userland pieces that could take advantage of them.

On most systems, people won't do that, they'll just use whatever kernel headers and glibc binaries their distro ships. If they want to change any of these, they have to know what they're doing.

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#124 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Matt Mackall

So we follow dhowell's plan with the following additions:

No.

We do _not_ move stuff over that is questionable.

Umm. I don't think you even read what I wrote.

Quoted 2 lines

I thought that was clear by now. The rules are:
- we only move things that _have_ to move

Yes. And when we move a definition from linux/foo.h to user/foo.h, we include them from linux/foo.h..

Quoted 2 lines

- we don't break existing programs, and no "but they are broken already" is not an excuse.

..which prevents userland breakage during the transition (and is generally what you want in the kernel anyway).

Quoted 2 lines

- we only move things where that _particular_ move can be shown to be beneficial.

It's my opinion that it will eventually be deemed beneficial to separate out everything that has a legitimate reason to be used by userspace. But I'm not suggesting getting there in one go, what I'm suggesting is how to get there incrementally. To rehash:

1. create include/user and friends
2. when we run across a troublesome ABI definition:
create include/user/foo.h and move the definition there make sure include/linux/foo.h includes it
userspace and kernel compile as before
send a patch
3. repeat step 2 as often as useful
4. add new user ABI always to include/user/
5. if at some point we find that all of userspace builds from include/user/ without reference to include/linux/, declare include/user the canonical ABI
6. drop #define __KERNEL__, etc..
7. drop any superfluous include/linux -> include/user includes (there shouldn't be many)

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#81 lore raw
In reply to Alexandre Oliva
Quoted 3 lines · Alexandre Oliva

Sure, we could take headers from linux-*/include/user and install them in /usr/include/kernel, but then includes in there that reference other headers in user/ or in asm-<arch>/ will cease to work.

I covered this with an evil sed script earlier in the thread.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#121 lore raw
In reply to Chris Friesen
Quoted 7 lines · Chris Friesen
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines

b) when include/user is deemed sufficiently populated, a flag day is declared and links from /usr/include are switched to them

there are no such links, only copies (more or less modified)

Indeed.

Quoted 4 lines

This may be somewhat heretical, but someone has to ask...

Once include/user/foo.h is sufficiently clean and sufficiently complete, is there any reason to not allow such links?

Briefly it's preferable not to need the kernel source nor a particular version of the kernel source around to compile things. Though a complete include/user makes building a proper kernel-headers package to install in /usr/include pretty trivial.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#112 lore raw
In reply to Al Viro
Quoted 5 lines · Al Viro
Quoted 2 lines

Wrong. These dirs must be linked to /usr/include so they must have more meaningfull names.

WTF? I've got a dozen kernel trees hanging around, which one (and WTF any, while we are at it) should be "linked to"?

Linked, copied, mount --binded, whatever. Just not under the name /usr/include/user, but something more meaningfull.

And to directly answer your question: you'll 'link' the newest kernel you've got around (we all know linux doesn't break abi compatibility, so the newest kernel will be fully compatible with all other kernel lying around, right? :)

Signature · 5 lines
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
#116 lore raw
In reply to Al Viro
Quoted 7 lines · Al Viro
Quoted 1 line · Mariusz Mazur

Wrong. These dirs must be linked to /usr/include so they must have more meaningfull names.

WTF? I've got a dozen kernel trees hanging around, which one (and WTF any, while we are at it) should be "linked to"?

Just my point of view, but I'd love to have /usr/include/linux and /usr/include/asm linked to the headers for the currently running kernel.

For a while I was playing with adding a bunch of new syscalls, and it sure would have been nice to be able to just include <asm/unistd.h> and get the up-to-date syscalls for whatever kernel I happened to be running.

I've since switched to using a device node and doing ioctl() calls to get around this, but the issue still stands as I need to either import the ioctl() command numbers and data types, or else I have to duplicate the header in userspace and keep the two versions up to date.

Chris

#85 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds

The way to prevent _future_ abuse is by adding something like

	#ifndef __KERNEL__
	#warning "This really doesn't work"
	#endif

which does that, and has the advantage of not breaking anything at all.

In other words: if you want to move things around just to break things, THEN THAT IS INCREDIBLY STUPID. We don't do things to screw our users over.

Linus, you're arguing that it's better to let users use something which is non-portable and silently does the wrong thing, as long as it actually compiles. That this is preferable to making sure it doesn't compile.

Now, there are cases (like _perhaps_ byteorder.h) where we should probably allow this kind of 'abuse' to continue because it's fairly harmless and it does actually _work_.

But atomic.h isn't an example of that.

If I wrote a userspace program which relies upon the md5sum of certain kernel headers not changing, would you decree that the headers in question should not change at _all_ because that would 'break existing software'? Of course not; you have to draw the line somewhere. And I would draw it somewhere between atomic.h and byteorder.h -- where would _you_ draw it?

Signature · 1 line
dwmw2
#126 lore raw
In reply to Linus Torvalds
Quoted 16 lines · Linus Torvalds
Quoted 2 lines · Matt Mackall

So we follow dhowell's plan with the following additions:

No.

We do _not_ move stuff over that is questionable.

I thought that was clear by now. The rules are:
- we only move things that _have_ to move
- we don't break existing programs, and no "but they are broken already" is not an excuse.
- we only move things where that _particular_ move can be shown to be beneficial.

Quoted 4 lines

No whole-sale moves. No "let's break things that I think are broken". No "let's change things because we can".

Well-defined moves. Both in content _and_ in reason.

Fine. And yes, we do it piecemeal with each part being given individual consideration.

I'd like to aim for a conclusion where everything which userspace needs is in the new directories and nothing is required from the private directories any more. But let's start with the really important parts which really _are_ crying out to be fixed.

Signature · 1 line
dwmw2
#86 lore raw
In reply to David Woodhouse
Quoted 5 lines · David Woodhouse

Linus, you're arguing that it's better to let users use something which is non-portable and silently does the wrong thing, as long as it actually compiles. That this is preferable to making sure it doesn't compile.

I'm saying that if it worked before, it should work after. And my suggestion gives a nice warning.

Quoted 1 line

But atomic.h isn't an example of that.

Even atomic.h. I could well imagine that somebody includes atomic.h just to get the thread-safe updates for some architectures. For example, asm-alpha/atomic.h does it right, and I woul dnot be at all surprised if somebody had noticed.

And your suggestion has the problem that the people who get bitten by a non-compiling thing are not necessarily the same people who can fix it.

Quoted 3 lines

software'? Of course not; you have to draw the line somewhere. And I would draw it somewhere between atomic.h and byteorder.h -- where would _you_ draw it?

I'll draw it at "somebody might validly use it", because the fact is, I can't test it.

Which is why I want patches to this to be OBVIOUSLY CORRECT, dammit! How hard is that to understand?

The fact is, the less benefit there is from a change, the more obviously correct it has to be in all forms. Moving stuff around in header files basically fixes zero bugs in the kernel, so the benefit for the kernel is basically zero. Which means that the obviousness factor of the change had better be pretty close to infinite.

Comprende?

Linus

#122 lore raw
In reply to Chris Friesen

In article <[email protected]>,

Quoted 6 lines · Chris Friesen
Quoted 4 lines · Christoph Hellwig
Quoted 2 lines

b) when include/user is deemed sufficiently populated, a flag day is declared and links from /usr/include are switched to them

Quoted 6 lines
Quoted 1 line

there are no such links, only copies (more or less modified)

This may be somewhat heretical, but someone has to ask...

Once include/user/foo.h is sufficiently clean and sufficiently complete, is there any reason to not allow such links?

It would be cleaner to use gcc -I /lib/modules/uname -r/build/include so that you don't break regular compiles by accident.

Mike.

#88 lore raw
In reply to David Woodhouse
Quoted 4 lines · David Woodhouse

The concept isn't at all hard to understand. But no patch is 'obviously correct' if you want to protect against the _slightest_ possibility that people might be abusing something you're taking away.

I really disagree. That's kind of my point. We _can_ make sure that there is abzolutely zero semantic content change.

People both inside and outside the kernel who use the old <linux/xxx.h> headers will get exactly what they got before if we do it right.

Quoted 3 lines

Some people might define __KERNEL__ on purpose when compiling something in userspace, to get something that would otherwise be hidden from them. Would you consider that sacrosanct too?

Why _do_ you want to break things? Do the cleanup. Don't do the breakage.

Linus

#87 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

I'll draw it at "somebody might validly use it", because the fact is, I can't test it.

Which is why I want patches to this to be OBVIOUSLY CORRECT, dammit! How hard is that to understand?

The concept isn't at all hard to understand. But no patch is 'obviously correct' if you want to protect against the _slightest_ possibility that people might be abusing something you're taking away.

Some people might define __KERNEL__ on purpose when compiling something in userspace, to get something that would otherwise be hidden from them. Would you consider that sacrosanct too?

Signature · 1 line
dwmw2
#94 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

Even atomic.h. I could well imagine that somebody includes atomic.h just to get the thread-safe updates for some architectures. For example, asm-alpha/atomic.h does it right, and I would not be at all surprised if somebody had noticed.

In fact, this is not entirely theoretical. I know people _have_ noticed, because I've gotten queries from some projects that wanted to copy the definitions for their alpha port. I only got those queries because my name is on the file, and those people wanted to actually copy them, not just include the header file. I don't know _how_ many people decided to just do the #include.

There are probably more people familiar with the kernel source tree than with GLIB which is probably the preferred way to do those things these days. And a few years ago that choice wasn't even there.

Linus

#89 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · David Woodhouse

The concept isn't at all hard to understand. But no patch is 'obviously correct' if you want to protect against the _slightest_ possibility that people might be abusing something you're taking away.

I really disagree. That's kind of my point. We _can_ make sure that there is abzolutely zero semantic content change.

We've _never_ made sure that there's absolutely zero semantic content change in our private headers.

Quoted 2 lines

People both inside and outside the kernel who use the old <linux/xxx.h> headers will get exactly what they got before if we do it right.

They'll get that if we change nothing at all.

Quoted 5 lines
Quoted 3 lines

Some people might define __KERNEL__ on purpose when compiling something in userspace, to get something that would otherwise be hidden from them. Would you consider that sacrosanct too?

Why _do_ you want to break things? Do the cleanup. Don't do the breakage.

I'm trying to understand precisely where _you_ want to draw the line between 'cleanup' and 'breakage'.

You obviously don't agree that including atomic.h was an entirely stupid thing for userspace to be doing in the first place, and that it's reasonable for that not to compile in future; you believe that it should continue to compile and silently do the wrong thing. We've covered that much.

So where _do_ you draw the line? Trying to use spinlock.h? Defining __KERNEL__ before including kernel headers?

I can try to apply common sense here, having discussed it with a bunch of other people. But you don't seem to agree with that; hence further guidance would be appreciated. But we can just submit patches to do the cleanups and see what you take and what you don't, if you really want to make us jump through hoops for the sake of it without indicating what you'd accept and what you'd not beforehand.

Signature · 1 line
dwmw2
#90 lore raw
Quoted 6 lines · David Woodhouse
Quoted 3 lines

I really disagree. That's kind of my point. We _can_ make sure that there is abzolutely zero semantic content change.

We've _never_ made sure that there's absolutely zero semantic content change in our private headers.

Not true. For example, I don't take spelling fixes that do semantic content changes.

See? The more trivial the fix, the more obviously correct it has to be.

This isn't even a "fix". It's a cleanup. It goes under the same rules a spelling fix does.

Linus

#91 lore raw
Quoted 2 lines · Linus Torvalds

This isn't even a "fix". It's a cleanup. It goes under the same rules a spelling fix does.

So you don't see a long-term technical benefit in cleaning up the API/ABI we export to userspace so that userspace stops depending on stuff which just isn't supposed to be there? It's all just cosmetic masturbation as far as you're concerned? There's no point in trying to get to the point where we don't need to separately maintain a glibc-kernheaders package because it can be taken directly from the kernel?

Signature · 1 line
dwmw2
#92 lore raw
Quoted 8 lines · David Woodhouse
Quoted 2 lines · Linus Torvalds

This isn't even a "fix". It's a cleanup. It goes under the same rules a spelling fix does.

So you don't see a long-term technical benefit in cleaning up the API/ABI we export to userspace so that userspace stops depending on stuff which just isn't supposed to be there?

I see _benefits_, but exactly because they aren't immediate, I think we should be careful.

Also, exactly because moving things around is a total _nightmare_ from a diff and source maintenance perspective, we should be _extra_ careful. It's almost impossible to see something that changed when that something is _also_ being moved around. You don't see it in the diff, and you don't see it in any of the tools - a small semantic change that would have been absolutely OBVIOUS if it was in a regular diff gets totally drowned out by the "move" diff.

Put another way: when people do things like re-indenting a driver (which has _exactly_ the same issues: cleanup for the future, but totally destroys all chance of seeing what the changes actually were), I want the patch that does the re-indentation to be totally independent of the actual changes.

Ie we do one patch that _only_ does the whitespace changes, and try to make sure that it doesn't break anything even by mistake. People even compile the thing and verify that it is 100% the same thing as an object file. Then that gets applied. Only _after_ that should you make changes.

Do we follow this religiously? For small things, clearly no. We sometimes mix whitespace and real changes. But we sure as hell shouldn't. At least not for anything bigger.

The EXACT same thing is true of any header file movement. DO NOT CHANGE SEMANTICS WHEN YOU MOVE STUFF!

It's not just my personal hangup. It's a damn good idea.

I will hereby just ignore this thread. Please remove me from the Cc, I've wasted way too much time on this total idiocy already. I've made my opinions extremely clear, and quite frankly, I don't want to hear anything on this any more for at least a week. It's just not worth it.

Linus

#96 lore raw

cdrom.h (was Re: [RFC] Splitting kernel headers...)

In reply to David Woodhouse
Quoted 3 lines · David Woodhouse

Now, there are cases (like _perhaps_ byteorder.h) where we should probably allow this kind of 'abuse' to continue because it's fairly harmless and it does actually _work_.

OK, time for my pet peeve: linux/cdrom.h.

#include <asm/byteorder.h>

[...]

/* for CDROM_PACKET_COMMAND ioctl */
struct cdrom_generic_command
{
[...]

        struct request_sense    __user *sense;

[...]
}

struct request_sense {

#if defined(__BIG_ENDIAN_BITFIELD)
        __u8 valid              : 1;
        __u8 error_code         : 7;
#elif defined(__LITTLE_ENDIAN_BITFIELD)
        __u8 error_code         : 7;
        __u8 valid              : 1;
#endif
        __u8 segment_number;

[...]
}

At least one architecture's asm/byteorder.h has been C++-incompatible in the past (I forget what; it was 2 years ago that I cared about it). So to fix this, we need a asm-*/user/byteorder.h that only describes the endianness. Except ... what about mips/mipsel?

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#113 lore raw
In reply to Mariusz Mazur
Quoted 9 lines · Mariusz Mazur
Quoted 5 lines · Al Viro
Quoted 2 lines

Wrong. These dirs must be linked to /usr/include so they must have more meaningfull names.

WTF? I've got a dozen kernel trees hanging around, which one (and WTF any, while we are at it) should be "linked to"?

Linked, copied, mount --binded, whatever. Just not under the name /usr/include/user, but something more meaningfull.

Whats wrong with
/lib/modules/uname -r/source/include/user
/lib/modules/uname -r/source/include/$arch

Sam

#117 lore raw
In reply to Linus Torvalds
Followup to:  <[email protected]>
By author:    Linus Torvalds <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 4 lines

But claiming that __KERNEL__ doesn't work is clearly a bunch of crapola. As is the notion that you can somehow do it all. We do it in small pieces.

IMNSHO, based on my experience with klibc, the problem is not #ifdef __KERNEL__, the *specific* problems are:

- The kernel exporting libc4/5 internals, which are neither what the kernel nor any kind of modern userspace wants. This should be possible to get rid of easily enough.

- The kernel exporting things into the userspace namespace. The kernel's definition of "struct stat" isn't usable -- the one that is useful is called "struct stat64" on most, but not all, architectures, but because it's a structure tag it can't be pulled from the exported kernel ABI in such a way that what userspace calls "struct stat" is what the kernel currently calls "struct stat64" if that exists.

This one is a bit uglier, since it probably needs something like:

  #ifdef __KERNEL__
  #define __kabi_stat64 stat64
  #endif

struct __kabi_stat64 { ... };

-hpa

#93 lore raw
In reply to David Woodhouse
Quoted 11 lines · David Woodhouse
Quoted 2 lines · Linus Torvalds

This isn't even a "fix". It's a cleanup. It goes under the same rules a spelling fix does.

So you don't see a long-term technical benefit in cleaning up the API/ABI we export to userspace so that userspace stops depending on stuff which just isn't supposed to be there? It's all just cosmetic masturbation as far as you're concerned? There's no point in trying to get to the point where we don't need to separately maintain a glibc-kernheaders package because it can be taken directly from the kernel?

There obviously _is_ a technical benefit. And to put out second usual argument used to reject patches, this benefit is desirable/matters to more than single digit number of users.

But lets face it, the changes you guies are discussing in this long thread are _massive_. May be not so much theorotical mass. In theory this is just a trivial movement of text from file to file with new rules enforced on adding new text.
But this does boil down to a great number of user interfaces being touched. And as far as the linux engineering practices go, this is _huge_.

The costs are high, each header file change is going to affect everything, the regression testing is going to be pain if we actually go do the modifications suggested by this thread.

And lets face it, with a retard[1] chosen as our leader , _mistakes will happen_. Things will _break_ and any decision taken will more often be a false one.

But we already know a better way to do things. After all thats how we have taken linux to the place where it is today. We can just start making the changes. In small steps. See if anyone is really bothered by the change. See if enough people give you feedback so that you are bothered to propagate the change to newer level.

I would much rather like to see the glibc-kernelheaders shrunk by few bytes every 3-4 months or so. And yes this actually has a chance that it may never happen.

The whole "put forward an idea get a consensus" and then "try to engineer it" works well for theory. But not in practise. In the end, only the good things prevail not because they were created good, but more because bad things can't servive. Leave the whole idea of "putting a research together and try to do the ultimate right" for the FreeBSD team.

And don't try to do the ultimate right without any reasearch, thats microsoft's domain.

In summary, splitting the headers could be a good idea. But to have any acceptance you need to prove it by practical examples. And when you do new things, be prepared to accept that they might fail in practise.

BAIN

[1]see:
http://marc.free.net.ph/message/20041109.161141.2f4e1246.html

#95 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds

Even atomic.h. I could well imagine that somebody includes atomic.h just to get the thread-safe updates for some architectures. For example, asm-alpha/atomic.h does it right, and I would not be at all surprised if somebody had noticed.

In fact, this is not entirely theoretical.

indeed it's not. THere were projects that used the x86 atomic.h header.... but failed to notice that its not actually using the lock prefix if you don't define CONFIG_SMP. Now it's purely up to luck if the userspace headers in /usr/include have CONFIG_SMP defined or not, so there have been cases where people compiled on an UP box and got racy code. Was a production level project; normally they built on SMP but once they built on UP .. poof ;)

so atomic.h is *dangerous* for userspace to use, at least on x86. It gives a false feeling of atomicity.

So dangerous that for the RH/Fedora distros, we don't ship the questionable parts of it.

#114 lore raw
In reply to Sam Ravnborg
Quoted 6 lines · Sam Ravnborg
Quoted 2 lines

Linked, copied, mount --binded, whatever. Just not under the name /usr/include/user, but something more meaningfull.

Whats wrong with
/lib/modules/uname -r/source/include/user
/lib/modules/uname -r/source/include/$arch

Those are supposed to be userland-only headers that don't just change - they are gradually expanded. I don't see a point in having uname -r in there.

And another thing - distribution vendors will *hate* anyone, that encourages app developers to add an include path based on which kernel is being currently run. People, that have their headers tied to their kernels are a *minority*.

(though iirc eg. netfilter is a mess, so there might be bigger reasons to ignore my arguments; not that I won't hate you if you do, and I'll have to patch everything, just to get it compiled)

Signature · 5 lines
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
#54 lore raw
In reply to Al Viro

Hi,

Quoted 8 lines · Al Viro

That leaves affs_fs.h containing only AFFS_SUPER_MAGIC declaration and amigaffs.h also seriously cleaned up; it still has a bunch of defines that are almost certainly never used in userland (they refer to ->b_data and in-core affs superblock), but these might in theory be useful (similar stuff for ext2 *is* used by userland code).

If Roman is OK with their removal, AFFS_DATA and friends should also get moved to fs/affs/affs.h.

Looks good. Thanks.
I wouldn't mind moving it all to fs/affs, I don't think there is any userspace dependency here. I have a simple mkaffs tool, but that uses a copy of it.

fs/affs/affs.h line 47

Quoted 8 lines
fs/affs/affs.h +8 -0
+#if 0
+	s32	 i_original;			/* if != 0, this is the key of the original */
+	u32	 i_data[AFFS_MAX_PREALLOC];	/* preallocated blocks */
+	int	 i_cache_users;			/* Cache cannot be freed while > 0 */
+	unsigned char i_hlink;			/* This is a fake */
+	unsigned char i_pad;
+	s32	 i_parent;			/* parent ino */
+#endif

That can be killed.

bye, Roman

#45 lore raw
In reply to David Woodhouse
Quoted 10 lines · David Woodhouse

I've lost track of the number of times things have broken because of incorrect use of kernel headers from userspace. That's what we're trying to fix -- by putting only the bits which are _supposed_ to be visible into files which userspace sees, where we know they define part of the userspace API and hence we can be extremely careful when editing them.

I don't think it makes sense at this point for us to bury our collective heads in the sand and pretend there isn't a problem here that's worth fixing.

Quoted 5 lines

I agree that it should be obviously correct though -- and that's why we're trying to end up with a structure that in the first pass would give us in userspace essentially what we already have in the various glibc-kernheaders packages, but without the constant and unnecessary need for some poor sod to keep those up to date by hand.

The concept of copying kernel headers into applications is even worse when arch portability is affected. I stopped counting how often I had to fix that kind of crap - and the state of kernel headers and userspace kernel header packages is really provoking that kind of mess.

Ralf

#115 lore raw
In reply to Mariusz Mazur
Quoted 8 lines · Mariusz Mazur
Quoted 6 lines · Sam Ravnborg
Quoted 2 lines

Linked, copied, mount --binded, whatever. Just not under the name /usr/include/user, but something more meaningfull.

Whats wrong with
/lib/modules/uname -r/source/include/user
/lib/modules/uname -r/source/include/$arch

Quoted 7 lines

Those are supposed to be userland-only headers that don't just change - they are gradually expanded. I don't see a point in having uname -r in there.

And another thing - distribution vendors will *hate* anyone, that encourages app developers to add an include path based on which kernel is being currently run. People, that have their headers tied to their kernels are a *minority*.

Above shows how this minority could locate headers for running kernel. It is not meant to be the general solution.

Sam

#64 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Alexandre Oliva

On Nov 30, 2004, Linus Torvalds <[email protected]> wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

If you want to use a legal analogy, the ABI is not a contract, it's a public _license_.

I didn't mean to use a legal analogy. I meant contract in the software engineering sense. Sorry if that wasn't clear.

Then your definition of a "contract" is flawed or your world-view has nothing to do with reality.

It's not my definition, it's a definition used in software engineering. Sure, if you ask a legal mind, you'll get a different answer. If you ask a bridge player, you'll get yet another definition. That's why we have jargons in which common-use words have their meanings narrowed to the specifics of certain fields. And that's why it's so hard for specialists in different fields to talk to each other at times when there's disagreement about the precise meaning of certain words.

Quoted 1 line

An ABI is not a contract.

Not a legal contract, for sure.

An ABI is a definition of an interface, including operations with pre- and post-conditions, data structures with their invariants, constants, file formats, etc. Most of that is covered by the software engineering term `contract'.

Sure enough, the headers we're talking about describe only a very small portion on that; pretty much only data structures and constants. It certainly doesn't *specify* the entire ABI, it only highlights portions of it.

Anyhow, all of this is beyond the point. I see you've decreed that people can introduce `user' directories in the kernel now. Would you please reconsider and choose a dir name that would enable the same ABI headers to be used by kernel and userland, without adding a directory to /usr/include that has no indication that it comes from the kernel?

Thanks,

Signature · 3 lines
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
#65 lore raw
Quoted 5 lines · Alexandre Oliva

Anyhow, all of this is beyond the point. I see you've decreed that people can introduce `user' directories in the kernel now. Would you please reconsider and choose a dir name that would enable the same ABI headers to be used by kernel and userland, without adding a directory to /usr/include that has no indication that it comes from the kernel?

As I've said before, this can be fixed with a sed script.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#46 lore raw
In reply to Ralf Baechle
Followup to:  <[email protected]>
By author:    Ralf Baechle <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 6 lines

The concept of copying kernel headers into applications is even worse when arch portability is affected. I stopped counting how often I had to fix that kind of crap - and the state of kernel headers and userspace kernel header packages is really provoking that kind of mess.

Indeed. The major reason why we need the kernel to export its own ABI information is because there are at my last counting 18 architectures running stock Linux, and each one of them has *at least* one ABI, and they're all different.

-hpa

#66 lore raw
In reply to Alexandre Oliva
Quoted 6 lines · Alexandre Oliva
Quoted 3 lines

Then your definition of a "contract" is flawed or your world-view has nothing to do with reality.

It's not my definition, it's a definition used in software engineering.

Not that I've seen. Maybe your school. Google also doesn't seem to agree with you. In other words, it seems to be in pretty limited use.

Quoted 3 lines

An ABI is a definition of an interface, including operations with pre- and post-conditions, data structures with their invariants, constants, file formats, etc.

Oh, I know what an ABI is. I just don't think your "contract" part has anything to do with it.

Quoted 1 line

Most of that is covered by the software engineering term `contract'.

I think you're making that up. Maybe there's some sw cult that swears by "contract programming", the same way there are the "extreme programming" cults etc. For example, I find this "Design by Contract" cult for object- oriented programming, but it has _zero_ to do with external API's, and is all about the interfaces for object-oriented components.

IOW, I don't find your arguments or your language usage in the least convincing. But hey, I did all my CS stuff outside of the US, whatever.

Linus

#67 lore raw
Quoted 2 lines · In article <[email protected]> you wrote:

I think you're making that up. Maybe there's some sw cult that swears by "contract programming"

Design by Contract is the reason for descibing the agreement between caller/callee as an contract. Bertran Meyer added the pre-conditions and post-conditions (kind of asserts) to his Eiffel Language, and I dont think that that is limited to a single system, but is also valid for bondaries like an ABI. It describes conventions like syscall numbers, too.

Greetings
Bernd

PS: http://en.wikipedia.org/wiki/Eiffel_programming_language

#68 lore raw
Quoted 7 lines · Bernd Eckenfels
Quoted 1 line · In article <[email protected]> you wrote:

I think you're making that up. Maybe there's some sw cult that swears by "contract programming"

Design by Contract is the reason for descibing the agreement between caller/callee as an contract.

Design by contract isn't really an agreement between the caller and the callee (which may not even exist when the contract is created). It's more of a (one way) promise by the callee that if the interface is used as described in the contract that it will correctly perform the advertised operation (where the advertised operation generally includes descriptions of possible failures and how they will be signalled). Most C APIs meet these criteria even though their pre and post conditions are expressed less formally than an Eiffel interface.

Quoted 4 lines

Bertran Meyer added the pre-conditions and
post-conditions (kind of asserts) to his Eiffel Language, and I dont think that that is limited to a single system, but is also valid for bondaries like an ABI. It describes conventions like syscall numbers, too.

I agree but think it's important to realize that it's a unilateral promise on the part of the callee (rather than agreement between the callee and the caller) which is in accord with Linus's view of the ABI.

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#71 lore raw
In reply to Linus Torvalds

Hi Linus,

[I am getting off-topic but...]

Quoted 1 line · Alexandre Oliva
Quoted 1 line

Most of that is covered by the software engineering term `contract'.

Quoted 5 lines · Linus Torvalds

I think you're making that up. Maybe there's some sw cult that swears by "contract programming", the same way there are the "extreme programming" cults etc. For example, I find this "Design by Contract" cult for object- oriented programming, but it has _zero_ to do with external API's, and is all about the interfaces for object-oriented components.

How is a class interface different from an external API? Sure, you don't want to change a _published_ API too often but I fail to see how internal and external APIs are fundamentally different.

I think Alexandre's definition has in fact originated from the "Design by Contract" cult. And it's all pretty simple stuff:

- You need to call a function in a certain way (precondition). The
caller fills this part
of the contract. Now the caller can _expect_ something from the
function (see below).
- The function does what it has promised to do and optionally
returns some values
(postcondition). The callee fills this part of the contract.
- The function expects some parts of the system state to remain stable during execution (invariants). In the kernel, you use BUG_ON for these btw.

So there's nothing new here really. However, if your _tools_ support Design by Contract, you can be explicit about this and enforce the 'contract' during compile-time or run-time.

And I think you're already doing this with your "require spinlock to be taken" sparse thingy...

Quoted 2 lines · Linus Torvalds

IOW, I don't find your arguments or your language usage in the least convincing. But hey, I did all my CS stuff outside of the US, whatever.

I don't think it's an US thing. At the university you did your CS stuff, we (well at least I) use the term 'contract' pretty much the same way Alexandre does...

Pekka

#69 lore raw
In reply to Peter Williams
Quoted 1 line · Peter Williams

Design by contract isn't really an agreement between the caller and the callee (which may not even exist when the contract is created). It's more of a (one way) promise by the callee that if the interface is used as described in the contract that it will correctly perform the advertised operation (where the advertised operation generally includes descriptions of possible failures and how they will be signalled). Most C APIs meet these criteria even though their pre and post conditions are expressed less formally than an Eiffel interface.

Design by Contract, as seen in the Eiffel language, is not a one way promise, it's a contract between the caller and callee. Both sides have to fullfil their part of the contract, the caller has to make sure that
the input are valid, and (only) given that, the callee can/has to make
sure that the caller gets the right output.

BTW, Bertrand Meyer is one of my professors, I should know the Eiffel language... :)

Quoted 1 line

I agree but think it's important to realize that it's a unilateral promise on the part of the callee (rather than agreement between the callee and the caller) which is in accord with Linus's view of the ABI.

Whenever you have two sides/parties, you have to agree on _something_, otherwise you can't communicate, if you speak to someone, you have to choose a language in which to speak, that's the first agreement. In this situation you have two sides, the kernel and the userspace. Your first agreement is the syscall number, and then the arguments, the type and format of the arguments, etc. Both sides have to agree on those things.

tom

#72 lore raw
In reply to Pekka Enberg
Quoted 2 lines · Pekka Enberg

How is a class interface different from an external API?

It's different from an external API exactly because it's not "external".

I dunno about you, but the (admittedly few) explicitly object-oriented projects I've seen from the inside, the class interface is something that you consider internal to your project, and are willing to change.

In Linux, the VFS layer comes pretty close: it's really very much about object-oriented programming, except the implementation is explicit rather than left up to the programmer. And we have very explicit rules on how you're supposed to do things: some are even _documented_ (oh, horrors) in Documentation/filesystems/vfs.txt.

And yes, in this kind of environment I actually think "contract" makes some kind of sense, because it is an internal thing that _has_ come out of agreement on both sides, and where changes need to be agreed upon by both user and definition. It's not an a-priori interface, it's something where both sides are intimately involved.

Quoted 13 lines

I think Alexandre's definition has in fact originated from the "Design by Contract" cult. And it's all pretty simple stuff:

- You need to call a function in a certain way (precondition). The caller fills this part
of the contract. Now the caller can _expect_ something from the function (see below).
- The function does what it has promised to do and optionally returns some values
(postcondition). The callee fills this part of the contract.
- The function expects some parts of the system state to remain stable during execution (invariants). In the kernel, you use BUG_ON for these btw.

Quoted 6 lines

So there's nothing new here really. However, if your _tools_ support Design by Contract, you can be explicit about this and enforce the 'contract' during compile-time or run-time.

And I think you're already doing this with your "require spinlock to be taken" sparse thingy...

Yes. And I think it makes sense _internally_ for a project.

But pretty much by definition, that's not an ABI. That's an "internal interfaces" kind of thing.

Quoted 7 lines · Linus Torvalds
Quoted 2 lines

IOW, I don't find your arguments or your language usage in the least convincing. But hey, I did all my CS stuff outside of the US, whatever.

I don't think it's an US thing. At the university you did your CS stuff, we (well at least I) use the term 'contract' pretty much the same way Alexandre does...

It may also be a new thing. Dammit, when I started, we had USCD Pascal, and we were happy. None of this OO stuff ;)

Linus

#73 lore raw
Quoted 1 line · Pekka Enberg
Quoted 1 line

How is a class interface different from an external API?

Quoted 12 lines · Linus Torvalds

It's different from an external API exactly because it's not "external".

I dunno about you, but the (admittedly few) explicitly object-oriented projects I've seen from the inside, the class interface is something that you consider internal to your project, and are willing to change.

In Linux, the VFS layer comes pretty close: it's really very much about object-oriented programming, except the implementation is explicit rather than left up to the programmer. And we have very explicit rules on how you're supposed to do things: some are even _documented_ (oh, horrors) in Documentation/filesystems/vfs.txt.

Quoted 5 lines

And yes, in this kind of environment I actually think "contract" makes some kind of sense, because it is an internal thing that _has_ come out of agreement on both sides, and where changes need to be agreed upon by both user and definition. It's not an a-priori interface, it's something where both sides are intimately involved.

I tend to think the other way around actually. For published (external) APIs, contracts are far more important because you cannot review or test the callers. For internal APIs, contracts can be more relaxed as you usually have full control over the codebase. So no, I don't think contracts are for internal use only.

Quoted 19 lines · Pekka Enberg
Quoted 19 lines

I think Alexandre's definition has in fact originated from the "Design by Contract" cult. And it's all pretty simple stuff:

- You need to call a function in a certain way (precondition). The caller fills this part
of the contract. Now the caller can _expect_ something from the function (see below).
- The function does what it has promised to do and optionally returns some values
(postcondition). The callee fills this part of the contract.
- The function expects some parts of the system state to remain stable during execution (invariants). In the kernel, you use BUG_ON for these btw.

So there's nothing new here really. However, if your _tools_ support Design by Contract, you can be explicit about this and enforce the 'contract' during compile-time or run-time.

And I think you're already doing this with your "require spinlock to be taken" sparse thingy...

Quoted 4 lines · Linus Torvalds

Yes. And I think it makes sense _internally_ for a project.

But pretty much by definition, that's not an ABI. That's an "internal interfaces" kind of thing.

And why not externally too? There are elements of a contract, I think. The kernel, for example, dictates that you must invoke syscall A with a pointer X pointing to an memory region of size E. Now, if the userland follows that contract, it gets service; otherwise it won't. And as soon as the userland _agrees_ with the contract, the kernel _must_ deliver the service as specified by the contract.

Please note that the kernel may change the contract on its part any time it wishes to do so. The userland can either accept the fact or reject it and live without the service. And this is what you're essentially doing right now. The only thing you're missing is actually _enforcing_ those contracts.

But anyway, I think I'm twisting the original definition of Design by Contract here for my own purposes and the only thing I violently disagreed with you was the fact that you claimed Alexandre was making up terms. ;)

Quoted 3 lines · Pekka Enberg
Quoted 3 lines

I don't think it's an US thing. At the university you did your CS stuff, we (well at least I) use the term 'contract' pretty much the same way Alexandre does...

Quoted 2 lines · Linus Torvalds

It may also be a new thing. Dammit, when I started, we had USCD Pascal, and we were happy. None of this OO stuff ;)

Yeah, they've ruined it completely now. They're pushing Java right from day one to youngsters :-)

Pekka

#70 lore raw
In reply to Tomas Carnecky
Quoted 11 lines · Tomas Carnecky
Quoted 1 line · Peter Williams

Design by contract isn't really an agreement between the caller and the callee (which may not even exist when the contract is created). It's more of a (one way) promise by the callee that if the interface is used as described in the contract that it will correctly perform the advertised operation (where the advertised operation generally includes descriptions of possible failures and how they will be signalled). Most C APIs meet these criteria even though their pre and post conditions are expressed less formally than an Eiffel interface.

Design by Contract, as seen in the Eiffel language, is not a one way promise, it's a contract between the caller and callee. Both sides have to fullfil their part of the contract, the caller has to make sure that
the input are valid, and (only) given that, the callee can/has to make
sure that the caller gets the right output.

BTW, Bertrand Meyer is one of my professors, I should know the Eiffel language... :)

Yes, but I still think it's essentially a one way thing. The writer of the callee doesn't have to consult with the writer of the caller to negotiate the contract (although that option isn't ruled out either) so it's essentially a unilateral promise on the part of the callee. This would especially be the case when writing library interfaces as it's impossible for the writer of the callee to negotiate with all the (potential) writers of callers to agree on an interface.

Quoted 6 lines
Quoted 2 lines

I agree but think it's important to realize that it's a unilateral promise on the part of the callee (rather than agreement between the callee and the caller) which is in accord with Linus's view of the ABI.

Whenever you have two sides/parties, you have to agree on _something_, otherwise you can't communicate, if you speak to someone, you have to choose a language in which to speak, that's the first agreement. In this situation you have two sides, the kernel and the userspace. Your first agreement is the syscall number, and then the arguments, the type and format of the arguments, etc. Both sides have to agree on those things.

Not really. In the final analysis, userspace has to accept what the kernel provides. It's not really a conversation it's more like a billboard on which the kernel describes an interface and makes promises about how it will operate if used according to the specification (or contract). If the caller fails to use the interface within the constraints of the specification (or contract) then the result is indeterminate and the callee can do whatever it likes including failing silently or even crashing (although that wouldn't be a good idea for a kernel interface).

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#125 lore raw
In reply to Matt Mackall
Quoted 3 lines · Matt Mackall

It's my opinion that it will eventually be deemed beneficial to separate out everything that has a legitimate reason to be used by userspace.

That's quite obvious and the whole thread is all about just that.

Quoted 1 line

But I'm not suggesting getting there in one go,

As this is impossible, given du -s /usr/src/linux/include/*.

Quoted 9 lines

what I'm
suggesting is how to get there incrementally. To rehash:

1. create include/user and friends
2. when we run across a troublesome ABI definition:
create include/user/foo.h and move the definition there make sure include/linux/foo.h includes it userspace and kernel compile as before
send a patch

What _is_ a "troublesome ABI definition" and who would qualify it as such?

Quoted 1 line

3. repeat step 2 as often as useful

Who would do that?

Quoted 1 line

4. add new user ABI always to include/user/

Who would watch that? How do you differentiate between a user API/ABI and an internal kernel API, if you don't know the code in question at all?

Quoted 2 lines

5. if at some point we find that all of userspace builds from include/user/ without reference to include/linux/,

How can we find that? Compiling the Fedora Code 7 would not be enough.

Quoted 2 lines

7. drop any superfluous include/linux -> include/user includes (there shouldn't be many)

Why not? Kernel "private" headers can and must include ABI headers as the private .c files do.

This won't work. include/user won't work either. There can't be any "transition periods" nor "instant switchovers".

I can see only one way to solve the problem - gradually, without "transition periods" etc.

1. We leave existing include/linux and include/asm (and possibly asm-generic etc) in place.
2. A new directory(ies) such as include/kernel (internal kernel headers) is created.
3. Code authors, who (I hope) know their code move the things they know are kernel-internal to the new directory(ies).
4. Userspace is explicitly forbidden to include anything in the new directory. We can even use some #ifndef __KERNEL__ #error user breakage as a tip.

In a year or two we can probably begin hunting remnants of the internals in include/linux and include/asm.

We can now consider anything in include/{linux,asm} which prevents userspace compilation etc. a bug. But bugs must be fixed by people who know how to fix them correctly.

Signature · 1 line
Krzysztof Halasa
#36 lore raw
In reply to Jeff Garzik
Quoted 17 lines · Jeff Garzik
Quoted 10 lines · Linus Torvalds

In short: having the kernel use the same names as user space is ACTIVELY BAD, exactly because those names have standards-defined visibility, which means that the kernel _cannot_ use them in all places anyway. So don't even _try_.

On the bigger question of what to do with kernel headers in general, let's just make one thing clear: the kernel headers are for the kernel, and big and painful re-organizations that don't help _existing_ user space are not going to happen.

Quoted 5 lines

In particular, any re-organization that breaks _existing_ uses is totally pointless. If you break existing uses, you might as well _not_ re- organize, since if you consider kernel headers to be purely kernel- internal (like they should be, but hey, reality trumps any wishes we might have), then the current organization is perfectly fine.

Quoted 8 lines

I don't think any drastic reorganization is even necessary.

Mariusz Mazur <[email protected]> updates
http://ep09.pld-linux.org/~mmazur/linux-libc-headers/ for each 2.6.x kernel release. linux-libc-headers are the kernel headers, with all the kernel-specific stuff stripped out. i.e. userland ABI only. Not sure how many distros have started picking that up yet... I think Arjan said Fedora Core had, or would.

I've got a problem that mmazur's headers come down on one side of and Eric Andersen of the busybox project comes down on the other side of. How the heck do you implement losetup without including linux/loop.h?

The way both busybox and util-linux do it is the to block copy out lots of ugly crap, include linux/version.h, and have #ifdefs to fix up differences between known kernel versions. I'm serious. This is from busybox/libbb/loop.c, but you can get an even more obfuscated variant of it out of util-linux:

--------------

/* Grumble... The 2.6.x kernel breaks asm/posix_types.h
* so we get to try and cope as best we can... */
#include <linux/version.h>
#include <asm/posix_types.h>

#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0)
#define __bb_kernel_dev_t __kernel_old_dev_t
#elif LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
#define __bb_kernel_dev_t __kernel_dev_t
#else
#define __bb_kernel_dev_t unsigned short
#endif

/* Stuff stolen from linux/loop.h */

#define LO_NAME_SIZE        64
#define LO_KEY_SIZE         32
#define LOOP_SET_FD         0x4C00
#define LOOP_CLR_FD         0x4C01
#define LOOP_SET_STATUS     0x4C02
#define LOOP_GET_STATUS     0x4C03
struct loop_info {
    int                lo_number;
    __bb_kernel_dev_t  lo_device;
    unsigned long      lo_inode;
    __bb_kernel_dev_t  lo_rdevice;
    int                lo_offset;
    int                lo_encrypt_type;
    int                lo_encrypt_key_size;
    int                lo_flags;
    char               lo_name[LO_NAME_SIZE];
    unsigned char      lo_encrypt_key[LO_KEY_SIZE];
    unsigned long      lo_init[2];
    char               reserved[4];
};

----------------------

When I submitted a patch to clean it up on the busybox list (fixing a real compile problem I had with maszur's 2.6.6.0 kernel headers):
http://www.busybox.net/lists/busybox/2004-November/013128.html

I got this response:
http://www.busybox.net/lists/busybox/2004-November/013132.html

Quoted 5 lines

No no no no no no no no no!

Rule #1: Never ever include kernel header files in user space. Rule #2: See Rule #1
Rule #3: See Rule #2

Apparently, linux/version.h isn't considered a kernel header?

Somehow, including linux/loop.h is considered bad, but including linux/version.h and having version specific #ifdefs in our code to fix up a block copied header for the specific kernel versions we know about is LESS incestuous knowledge. (After all, the above will OBVIOUSLY work on BSD, is guaranteed to continue to compile without modification under the 2.8 kernel, and is in generally much cleaner than deleting all that crap and just including linux/loop.h.)

I still don't understand the reasoning here. Mazur's cleaned-up kernel headers include a cleaned up version of linux/loop.h because loop_info simply isn't defined anywhere else we can #include, and it's something you need in order to set up loop devices. It's a structure userspace and the kernel use to pass data back and forth, both need to include it. Any _new_ header file created won't exist on 2.4, but this header IS there on 2.4, and 2.2, and 2.6. Including linux/loop.h is the only thing that will _work_, both in current and in future versions.

How is this _supposed_ to be done?

Rob

#37 lore raw
Quoted 1 line · Rob Landley

How the heck do you implement losetup without including linux/loop.h?

Copy the util-linux sources.

Quoted 3 lines

The way both busybox and util-linux do it is the to block copy out lots of ugly crap, include linux/version.h, and have #ifdefs to fix up differences between known kernel versions. I'm serious.

Yes, a well-known problem. Util-linux has roughly

#include <linux/posix_types.h>
#include <linux/version.h>

#if LINUX_VERSION_CODE < KERNEL_VERSION(2,5,68)
#define my_dev_t __kernel_dev_t
#else
#define my_dev_t __kernel_old_dev_t
#endif

Here the struct has not changed, but the names for the types have changed. Thus, instead of looking at <linux/version.h> and <linux/posix_types.h> one could have a completely kernel-independent source with different defines for each architecture.

But, all that nonsense is needed only for the obsolete struct loop_info. Any new program should use struct loop_info64, and it has a clean definition:

struct loop_info64 {

        __u64              lo_device;                   /* ioctl r/o */
        __u64              lo_inode;                    /* ioctl r/o */
        __u64              lo_rdevice;                  /* ioctl r/o */
        __u64              lo_offset;
        __u64              lo_sizelimit;/* bytes, 0 == max available */
        __u32              lo_number;                   /* ioctl r/o */
        __u32              lo_encrypt_type;
        __u32              lo_encrypt_key_size;         /* ioctl w/o */
        __u32              lo_flags;                    /* ioctl r/o */
        __u8               lo_file_name[LO_NAME_SIZE];
        __u8               lo_crypt_name[LO_NAME_SIZE];
        __u8               lo_encrypt_key[LO_KEY_SIZE]; /* ioctl w/o */
        __u64              lo_init[2];

};

#38 lore raw

Err,

Isn't this WRONG.

Why is losetup considering the kernel at compile time?

Should it not be determining the kernel version at runtime and using the right structures then?
In which case it should have obtained a conceptual .../user/loop.h from a 2.4.x release and include a private copy If 2.6.x is nice enough to provide an explicit .../user/loop.h then take a private copy of that into the losetup tree.

(Agreed that busybox may want to shrink as much as possible so allow build flags to say --with-24x-abi --with-26x-abi)

David

Quoted 58 lines · Andries Brouwer
Quoted 3 lines · Rob Landley

How the heck do you implement losetup without including linux/loop.h?

Copy the util-linux sources.

Quoted 3 lines

The way both busybox and util-linux do it is the to block copy out lots of ugly crap, include linux/version.h, and have #ifdefs to fix up differences between known kernel versions. I'm serious.

Yes, a well-known problem. Util-linux has roughly

#include <linux/posix_types.h>
#include <linux/version.h>

#if LINUX_VERSION_CODE < KERNEL_VERSION(2,5,68)
#define my_dev_t __kernel_dev_t
#else
#define my_dev_t __kernel_old_dev_t
#endif

Here the struct has not changed, but the names for the types have changed. Thus, instead of looking at <linux/version.h> and <linux/posix_types.h> one could have a completely kernel-independent source with different defines for each architecture.

But, all that nonsense is needed only for the obsolete struct loop_info. Any new program should use struct loop_info64, and it has a clean definition:

struct loop_info64 {

       __u64              lo_device;                   /* ioctl r/o */
       __u64              lo_inode;                    /* ioctl r/o */
       __u64              lo_rdevice;                  /* ioctl r/o */
       __u64              lo_offset;
       __u64              lo_sizelimit;/* bytes, 0 == max available */
       __u32              lo_number;                   /* ioctl r/o */
       __u32              lo_encrypt_type;
       __u32              lo_encrypt_key_size;         /* ioctl w/o */
       __u32              lo_flags;                    /* ioctl r/o */
       __u8               lo_file_name[LO_NAME_SIZE];
       __u8               lo_crypt_name[LO_NAME_SIZE];
       __u8               lo_encrypt_key[LO_KEY_SIZE]; /* ioctl w/o */
       __u64              lo_init[2];

};
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#39 lore raw
Quoted 3 lines · David Greaves

Err,

Isn't this WRONG.

You did not read my mail, or you did not understand it.

Please write the patch for losetup to avoid looking at kernel source. If you after writing think that it is cleaner than the present setup, submit it.

Quoted 2 lines

Should it not be determining the kernel version at runtime and using the right structures then?

See - you do not understand at all.
The structures do not depend on the kernel version.

Andries

#40 lore raw

(Trimming linus from cc: list because he asked.)

Quoted 10 lines · Andries Brouwer
Quoted 3 lines · David Greaves

Err,

Isn't this WRONG.

You did not read my mail, or you did not understand it.

Please write the patch for losetup to avoid looking at kernel source. If you after writing think that it is cleaner than the present setup, submit it.

Avoiding looking at the kernel source for the darn structure would be relatively easy if the thing didn't vary by architecture. I agree that hardwiring into userspace code #ifdefs for ARE_WE_ON_ALPHA is approximately as ugly as hardwiring in #ifdefs for IS_THIS_2.6.

Both are ugly. This is why header files exist. I still don't understand why we're not supposed to use 'em when the alternative is building incestuous knowledge about the kernel version we're compiling against into our application.

Bluntlly, I don't understand why "#include <linux/version.h>" and several #ifdefs is an improvement on "#include <linux/loop.h>" with no #ifdefs.

Rob

#41 lore raw
Quoted 2 lines · Rob Landley

Bluntlly, I don't understand why "#include <linux/version.h>" and several #ifdefs is an improvement on "#include <linux/loop.h>" with no #ifdefs.

Look at the history of util-linux.

Of course it did the latter. Up to the point that broke. Then invented a silly workaround. Up to the point that broke. Then tried something else. Up to the point that broke. Then gave up including this particular header and made a private copy. Again, there is some painful history with the private copy. I showed the current version.

You see - util-linux advertises: all kernels, all archs, all libc versions since 4.6.27.
If the kernel improves today, it is too late for util-linux.

But yes, the include situation is painful.
It seems there is some recent progress. We'll see what happens.

Andries

#130 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds

Ok, this discussion has gone on for too long anyway, but let's make it easier for everybody. The kernel uses u8/u16/u32 because:

- the kernel should not depend on, or pollute user-space naming. YOU MUST NOT USE "uint32_t" when that may not be defined, and user-space rules for when it is defined are arcane and totally arbitrary.

Hmm, so you're predicting problems if user space includes something that uses uint32_t ? Wouldn't either of these work (descriptive file names for clarity):

1) include/linux-user/foo.h:

    #include <stdint.h>
    #include <linux-kernel-and-user-common/foo.h>

    include/linux/foo.h:
    #include <linux/stdint.h>
    #include <linux-kernel-and-user-common/foo.h>

    (And document that anything under linux-user may pull in a
    certain set of standard headers, such as stdint.h, maybe
    sys/types.h, etc.)

2) include/linux-user/foo.h:

    #ifndef __KERNEL__
    #include <stdint.h>
    #else
    #include <linux/stdint.h>  /* if we want this */
    #endif
    ...
    foo stuff (or some include)
    ...

    Idem. (Sort of the opposite of what we have today.)

3) just require that some stdint.h has been included before including linux-user/foo.h

For user space that absolutely insists on doing its own uint32_t, a refined variant of case 3 (i.e. detailing of what exactly is expected to be defined) or, in case 1, direct use of linux-kernel-and-user-common/foo.h along with the definition from case 3 should work.

There are other cases where a header file may pull in other definitions, e.g. for fcntl.h, POSIX decrees

| Inclusion of the <fcntl.h> header may also make visible all | symbols from <sys/stat.h> and <unistd.h>.

So the semantics of for cases 1 and 2 would be consistent with current POSIX practice.

What am I missing ?

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#178 lore raw
In reply to H. Peter Anvin
Quoted 2 lines · H. Peter Anvin

Since we can't use typedefs on
struct tags, I suggest:

Wouldn't this be a wonderful extension to ask from the gcc folks ? typedef mumble struct foo; /* or union, enum */
with "mumble" either a "struct bar" or a typedef thereof.

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#131 lore raw
In reply to Werner Almesberger
Quoted 8 lines · Werner Almesberger

Hmm, so you're predicting problems if user space includes something that uses uint32_t ? Wouldn't either of these work (descriptive file names for clarity):

1) include/linux-user/foo.h:

    #include <stdint.h>
    #include <linux-kernel-and-user-common/foo.h>

No. Because when you include <sys/ioctl.h> (which includes the <linux-user/foo.h>, the POSIX/SuS/whatever namespace rules say that YOU MUST NOT pollute the namespace with the names from stdint.h.

The user is supposed to see "int32_t" and friends _only_ if the user himself includes <stdint.h> or one of the very specific headers that is documented by the standard to include it.

Trust me. We are NOT going to use <stdint.h> in the kernel.

This is a common issue with namespace pollution. For example, this program is perfectly valid afaik (well, except for being _stupid_, but that's another issue):

	#include <stdio.h>

	const char *int32_t(int i)
	{
		return i ? "non-zero" : "zero";
	}

	int main(int argc, char **argv)
	{
		return printf("argc is %s\n", int32_t(argc));
	}

Get the picture? The user may _use_ names that are defined for the standard - like "strlen()" or "uint32_t" - for his own nefarious purposes, because they are _only_ defined if you include the right header files.

And trust me, the rules are really arcane. Not only do you have several standards, and several versions, you have various local rules too, ie gcc and glibc make up their own rules about things that depend on compiler flags etc.

And that is why a kernel header file must not define them, and by implication, cannot use these names.

Because the whole _point_ of having kernel header files that can be used in user-space is that they are helpers for things like <sys/ioctl.h> etc, that would just copy them.

If we are going to pollute the user-visible namespace in random ways, the whole discussion is pointless. At that point, glibc (or any other libc) can't use those API headers anyway.

Remember: the _biggest_ reason to make kernel headers available is not to user programs that want them, but to libc and friends.

Linus

#132 lore raw
Quoted 3 lines · Linus Torvalds

No. Because when you include <sys/ioctl.h> (which includes the <linux-user/foo.h>, the POSIX/SuS/whatever namespace rules say that YOU MUST NOT pollute the namespace with the names from stdint.h.

Hmm, the POSIX 1003.1 2004 edition (which includes SuS) wisely remains silent on the issue of sys/ioctl.h, but I see what you mean.

I think some of my confusion arises from the fact that there aren't many well-known interfaces that use uint32_t and friends to start with. Instead, they use ssize_t, off_t and such (which admittedly have similar problems, e.g. in cases where one can't simply get all of sys/types.h), or private inventions like u_int32_t and such.

Therefore, stdint.h types would mainly be used with new interfaces, or in intermediate definitions which are not themselves part of the interface. Of course, the latter would have to consider pollution issues.

Would you agree that the use of stdint.h types is acceptable for new interfaces, provided that these come with the warning that their use may pull in all of stdint.h ?

Quoted 4 lines

And trust me, the rules are really arcane. Not only do you have several standards, and several versions, you have various local rules too, ie gcc and glibc make up their own rules about things that depend on compiler flags etc.

If this is as unpredictable as you describe it, it would mean that also new interfaces which need to specify an exact integer size would require new sets of type names. So horrors like my_uint32_t, project-specific and of course conflicting definitions of ULONG (really really meaning 32 bit, at least sometimes), etc. would still be with us for a long time.

Let me add that I've happily used the standard integer names for a while, inside and outside of the kernel, so far without ill effects. Maybe I've just been lucky.

Quoted 2 lines

Remember: the _biggest_ reason to make kernel headers available is not to user programs that want them, but to libc and friends.

Okay, for me it's usually exactly the opposite :-) New tools that need to share fairly private interfaces with the kernel.

Thanks,
- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#144 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds

This is a common issue with namespace pollution. For example, this program is perfectly valid afaik (well, except for being _stupid_, but that's another issue):

	#include <stdio.h>

	const char *int32_t(int i)
	{
		return i ? "non-zero" : "zero";
	}

Actually this is not allowed in POSIX. _Any_ header may define any identifier ending with "_t".

Andreas.

Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#145 lore raw
Quoted 9 lines · Andreas Schwab
Quoted 6 lines · Linus Torvalds

This is a common issue with namespace pollution. For example, this program is perfectly valid afaik (well, except for being _stupid_, but that's another issue):

	#include <stdio.h>
Quoted 7 lines
Quoted 4 lines
	const char *int32_t(int i)
	{
		return i ? "non-zero" : "zero";
	}

Actually this is not allowed in POSIX. _Any_ header may define any identifier ending with "_t".

.. and that's POSIX.

It's not true for BSD_SOURCE, for example.

WHICH IS MY POINT, DAMMIT!

How hard is it to understand? There are millions of different standards, many of them without any standards body at all, and just plain standards by virtue of "that's how people have done it for years".

uint32_t is totally useless for the kernel. We should never ever use it.

Linus

#133 lore raw
In reply to Werner Almesberger
Quoted 4 lines · Werner Almesberger

Would you agree that the use of stdint.h types is acceptable for new interfaces, provided that these come with the warning that their use may pull in all of stdint.h ?

Not really.

Two reasons:
- The fact is, things are interdependent. And this thing is all about _avoiding_ implicit dependencies, and rules like "if you include _this_ particular header file, then you must include yyy.h first, or you can't do zzzz".
- uint32_t and friends aren't all that nice anyway. Quite frankly, there is nada, zilch, _zero_, reason to use them over the __u8/__u16/__u32. In contrast, the __u8/__u16/__u32 types have obvious advantages, ranging from the _lack_ of namespace collission issue to just plain consistency.

iow, using "uint32_t" and friends is _stupid_. The only fathomable argument for doing so would be consistency, but since BY DEFINITION we cannot be consistent if we use them (since we cannot and must not use them anywhere), that _single_ reason for using them is actually the biggest reason to NOT use them.

See my point?

Quoted 3 lines

If this is as unpredictable as you describe it, it would mean that also new interfaces which need to specify an exact integer size would require new sets of type names.

No. They require an OLD set of type names, that have existed for years in the kernel, and that are already used widely.

In other words, they require __u8/__u16/__u32/__u64. That _is_ the way to specify a sized type in kernel headers. It has been that way for years, it's extremely readable, it's consistent, and it's compact.

In other words, u8/u16/u32/u64 (and the signed versions: s8 and friends) _are_ the standard names in the kernel, and the __ versions have always existed alongside them for things like header files that have namespace issues.

So forget about that stupid abortion called "uint32_t" already. It buys you absolutely nothing.

Linus

#139 lore raw
In reply to Werner Almesberger
Quoted 4 lines · Werner Almesberger
Quoted 3 lines · Linus Torvalds

No. Because when you include <sys/ioctl.h> (which includes the <linux-user/foo.h>, the POSIX/SuS/whatever namespace rules say that YOU MUST NOT pollute the namespace with the names from stdint.h.

[...]

Quoted 4 lines

Therefore, stdint.h types would mainly be used with new interfaces, or in intermediate definitions which are not themselves part of the interface. Of course, the latter would have to consider pollution issues.

They _can't_ show up in interfaces unless you specify that stdint.h has to be included before them... and I'd awise against any needless interface fattening. Besides, using them in the kernel would then mean pulling in stdint.h there... and you get the same mess, the other way around ("What userspace headers are OK to pull in when compiling the kernel?"). Better don't.

[...]

Quoted 8 lines
Quoted 4 lines

And trust me, the rules are really arcane. Not only do you have several standards, and several versions, you have various local rules too, ie gcc and glibc make up their own rules about things that depend on compiler flags etc.

If this is as unpredictable as you describe it, it would mean that also new interfaces which need to specify an exact integer size would require new sets of type names.

That's what all the _u8 and such are. Note that any name starting with _ or __ is explicitly reserved by the standard (i.e., users should never use them), so they are (relatively) safe.

Quoted 3 lines

So horrors like my_uint32_t, project-specific and
of course conflicting definitions of ULONG (really really meaning 32 bit, at least sometimes), etc. would still be with us for a long time.

Better use stdint.h where possible.

Quoted 3 lines

Let me add that I've happily used the standard integer names for a while, inside and outside of the kernel, so far without ill effects. Maybe I've just been lucky.

No. You just haven't commited horrors like the ones Linus showed. No halfway sane C programmer would, but they are quite legal (and must work).

Quoted 5 lines
Quoted 2 lines

Remember: the _biggest_ reason to make kernel headers available is not to user programs that want them, but to libc and friends.

Okay, for me it's usually exactly the opposite :-) New tools that need to share fairly private interfaces with the kernel.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#140 lore raw
Quoted 6 lines · Horst von Brand
Quoted 5 lines

Therefore, stdint.h types would mainly be used with new interfaces, or in intermediate definitions which are not themselves part of the interface. Of course, the latter would have to consider pollution issues.

Quoted 6 lines

They _can't_ show up in interfaces unless you specify that stdint.h has to be included before them... and I'd awise against any needless interface fattening. Besides, using them in the kernel would then mean pulling in stdint.h there... and you get the same mess, the other way around ("What userspace headers are OK to pull in when compiling the kernel?"). Better don't.

Today we only pull in stdarg.h from userspace - actually a compiler dependent file. We shall not introduce anything to pull more stuff in from userspace. We have namespace issues enough - keeping the kernel out of userspace namespace is important.

Sam

#141 lore raw
Quoted 3 lines · Sam Ravnborg

Today we only pull in stdarg.h from userspace - actually a compiler dependent file.

Indeed. You'll notice that gcc doesn't even put stdarg.h in /usr/include, it's in the compiler-specific header file directory, usually something like /usr/lib/gcc-lib/<vendor>/<version>/include.

We used to compile with "-nostdinc" to make sure that you couldn't include user files even by mistake, but that was removed for some reason I can't for the life of me remember any more.

We probably should do it again. Something vaguely like

	CFLAGS += -nostdinc -I $(CC -print-file-name=include)

should do it (and still pick up "stdarg.h").

Linus

#142 lore raw
Quoted 3 lines · Linus Torvalds

We used to compile with "-nostdinc" to make sure that you couldn't include user files even by mistake, but that was removed for some reason I can't for the life of me remember any more.

??? We still do that, see NOSTDINC_FLAGS.

Andreas.

Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#143 lore raw
Quoted 2 lines · Andreas Schwab

??? We still do that, see NOSTDINC_FLAGS.

Ahh, yup, there is it. Goodie, I was just confused.

Linus

#134 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

iow, using "uint32_t" and friends is _stupid_. The only fathomable argument for doing so would be consistency,

I think consistency is a worthwhile goal in this case. I guess we can work towards consistency even without using uint32_t in kernel headers.

This still bears the risk that people will copy __u32 and friends from headers to then non-portable applications, but I guess that's something we just have live with.

Since C now finally has standard types with well-defined sizes, I think there's little reason for applications not to use these types. So peer pressure should eventually make everyone at least compatible.

A first step towards enabling converence would be to _guarantee_ that glibc types are _identical_ to the kernel types, e.g.

/usr/include/stdint.h:
#include <linux-user/types.h> /* only __u32 and such */

typedef __u32 uint32_t;
...

(If they just "happen to be the same", people may still not trust this, and will feel justified inventing their own schemes. And as ugly as uint32_t may look, it's still infinitely better than code where each library contributes its own set of sized integers ...)

Then, in a few years, when uint32_t and friends are considered as much a part of the C language as "int", we might be able to sed s/__u32/uint32_t/g the kernel, and obtain perfect consistency.

Does this sound reasonable ?

Quoted 2 lines

So forget about that stupid abortion called "uint32_t" already. It buys you absolutely nothing.

Now, what do we do with them when they are inside the kernel, far from any interfaces ? Live and let live ? Deprecate them early on January 1st, when nobody is watching ? :-)

Thanks,
- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#135 lore raw
Quoted 5 lines · Werner Almesberger
Quoted 2 lines

So forget about that stupid abortion called "uint32_t" already. It buys you absolutely nothing.

Now, what do we do with them when they are inside the kernel, far from any interfaces ? Live and let live ?

Yes. As long as they don't cause problems, I'm a big fan of "let developers make their own choices". I think that uint32_t and friends are totally useless in the kernel, but while I have strong opinions, I try to not force those opinions on others too much.

Quite frankly, it's unlikely to cause problems in practice. Having strong opinions and enjoying the occasional flame war is one thing. Trying to rail against other people who don't share those opinions and force them to change their code is a totally different thing.

[ It has absolutely nothing to do with me being lazy. Oh, no. I'm not
lazy. Much better to attribute it to some lofty goal like "live and let live". If the two just _happen_ to co-incide, all the better ;-]

Linus

#136 lore raw
In reply to Werner Almesberger
Quoted 6 lines · Werner Almesberger

A first step towards enabling converence would be to _guarantee_ that glibc types are _identical_ to the kernel types, e.g.

/usr/include/stdint.h:
#include <linux-user/types.h> /* only __u32 and such */

Quoted 7 lines

typedef __u32 uint32_t;
...

(If they just "happen to be the same", people may still not trust this, and will feel justified inventing their own schemes. And as ugly as uint32_t may look, it's still infinitely better than code where each library contributes its own set of sized integers ...)

Not sure about this. I think such types should be defined by the kernel, for it's use only. The libc (and any other userspace software) may probably use them (if they have to), but must not (re)define them.
The userspace should probably best use standard types. This is userspace to know if their uint32_t has standard meaning equivalent to __u32 or if it's actually a function name.

So i think we should have:

kernel-include/headers.h:
#define __u32 ...

struct some_structure {
       __u32 var;
}str1;

and then userspace.c (libc et al) should:

#include linux/headers.h

str1 v;
uint32_t variable = v.var;

or maybe with some type conversion:

long long v2 = v.var;
unsigned char v3 = v.var;

Signature · 1 line
Krzysztof Halasa
#137 lore raw
Quoted 4 lines · Krzysztof Halasa

Not sure about this. I think such types should be defined by the kernel, for it's use only. The libc (and any other userspace software) may probably use them (if they have to), but must not (re)define them.

That's not what should be happening there: if the kernel only uses __u32 and friends but never uint32_t et al. at an interface,

typedef __u32 uint32_t;

should be a perfectly safe thing for glibc's stdint.h to do.

Quoted 1 line

The userspace should probably best use standard types.

Yes.

Quoted 3 lines

This is
userspace to know if their uint32_t has standard meaning equivalent to __u32 or if it's actually a function name.

Yes, but that's decided by whether you include stdint.h or anything that has the effect of including stdint.h. What I'm saying is that there should be a guarantee that __u32 and uint32_t are always identical types. If they are merely assignment-compatible, you run into problems like this

/* somewhere in a kernel-to-glibc interface (on 32 bit) */ typedef unsigned int __u32;
...
static __u32 foo;

/* elsewhere, in glibc (on 32 bit) */
typedef unsigned long uint32_t;
...
static uint32_t bar;

/* finally, in the application */
if (&foo == &bar)

	/* do something */;

warning: comparison of distinct pointer types lacks a cast

Quoted 6 lines

and then userspace.c (libc et al) should:

#include linux/headers.h

str1 v;
uint32_t variable = v.var;

Interfaces are normally specified with the explicit data type of each item, in particular struct members. The masses will probably look for solid branches instead of olive twigs for you if you try to break this kind of assurance :-)

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#138 lore raw
Quoted 6 lines · Werner Almesberger

That's not what should be happening there: if the kernel only uses __u32 and friends but never uint32_t et al. at an interface,

typedef __u32 uint32_t;

should be a perfectly safe thing for glibc's stdint.h to do.

Aaah, right. I thought about #define there, it would be the other way around and thus wrong. Looks like I have to read my mail a bit earlier.

Signature · 1 line
Krzysztof Halasa