linux-libc-headers dead
Subsystems (2)
LLH hasn't seen a new release for a lot more than six months now and up until today I hoped to get back on track with new releases. But I've just spent some time doing a 2.6.14 update, and it came back to me, that I'd have to spend up to 10 hours just to get a basic 2.6.14.0 ready. And there'd still be 2.6.15 waiting, 2.6.16 just around the corner plus sorting through all the bug reports that came in during those months and all the internal rearranging I either had planned or that's being forced by new kernel releases (eg. addition of asm-powerpc).
I stopped having both the time and the will for such commitments a couple of months ago.
Should anyone want to take over, I'd be happy to give hints, pointers, and whatnot. Just don't get overexcited -- diffs between new kernel versions get bigger, not smaller, and after a couple of years there's still no long term solution in sight.
Oh, and women don't fall for the "I hack kernel stuff" line. I was lied to.
Signature · 5 lines
In the year eighty five ten God is gonna shake his mighty head He'll either say, "I'm pleased where man has been" Or tear it down, and start again
Hi,
Salı 14 Mart 2006 17:19 tarihinde şunları yazmıştınız:
Quoted 8 lines
LLH hasn't seen a new release for a lot more than six months now and up until today I hoped to get back on track with new releases. But I've just spent some time doing a 2.6.14 update, and it came back to me, that I'd have to spend up to 10 hours just to get a basic 2.6.14.0 ready. And there'd still be 2.6.15 waiting, 2.6.16 just around the corner plus sorting through all the bug reports that came in during those months and all the internal rearranging I either had planned or that's being forced by new kernel releases (eg. addition of asm-powerpc).
Thank you for all your work, its greatly appreciated. I think its time to fix public kernel headers so they can be included in userspace oh well I hear "Don't include kernel headers in userspace" flames already but some things are impossible without including kernel headers ( ide/v4l etc comes to mind ).
Regards,
ismail
Signature · 1 line
An eye for eye will make the whole world blind -- Gandhi
Quoted 9 lines
LLH hasn't seen a new release for a lot more than six months now and up until today I hoped to get back on track with new releases. But I've just spent some time doing a 2.6.14 update, and it came back to me, that I'd have to spend up to 10 hours just to get a basic 2.6.14.0 ready. And there'd still be 2.6.15 waiting, 2.6.16 just around the corner plus sorting through all the bug reports that came in during those months and all the internal rearranging I either had planned or that's being forced by new kernel releases (eg. addition of asm-powerpc).
Quoted 7 lines
I stopped having both the time and the will for such commitments a couple of months ago.
Should anyone want to take over, I'd be happy to give hints, pointers, and whatnot. Just don't get overexcited -- diffs between new kernel versions get bigger, not smaller, and after a couple of years there's still no long term solution in sight.
I'm interested, more so to be part of a team which as a group is responsible for linux-libc-headers. This way if one or two persons don't have much time available, the others can share the load.
If there is no-one else willing to maintain linux-libc-headers, I'll be more than happy to take it over and try get more developers interested.
Kind Regards
Signature · 1 line
Nigel Kukard, PhD CompSc
Quoted 4 lines
If there is no-one else willing to maintain linux-libc-headers, I'll be more than happy to take it over and try get more developers interested.
Some linux distributions (I know of Novell who do it for SUSE Linux) seem to roll their own thing AFAICS. The glibc.src.rpm from them contains a userspacified copy of the kernel headers.
Well, it's probably not bleeeeding edge uptodate, but provides a starting point.
Jan Engelhardt
Quoted 3 lines · Jan Engelhardt
Some linux distributions (I know of Novell who do it for SUSE Linux) seem to roll their own thing AFAICS. The glibc.src.rpm from them contains a userspacified copy of the kernel headers.
Yep. Red Hat does, too. They had to, since there wasn't a linux-libc-headers project when they started.
Maybe the way to move forward is to see if we can get the linux-libc-headers accepted as is into the kernel.org tree in a userspace-include directory, then maintain it there.
- Dan
--
Wine for Windows ISVs: http://kegel.com/wine/isv
Hi Mariusz :)
* Mariusz Mazur <[email protected]> dixit:
Quoted 2 lines
I stopped having both the time and the will for such commitments a couple of months ago.
I understand. And it's a pity, because LLH is very useful, and
you did a GREAT job with it.
Quoted 1 line
Oh, and women don't fall for the "I hack kernel stuff" line. I was lied to.
Yes... XDDD
Well, I can volunteer to test LLH and contribute a bit, fixing
things I can fix, but IMHO this kind of work should be done by the kernel itself: it should provide both user-space and kernel-space headers (yes, I know, "don't-include-kernel-headers-in-userspace". Tell that to iproute2, iptables, etc...).
Thanks a lot for your work, your effort and your interest in this
problem. You have done a lot for the linux community (at least, you've done a lot for me).
Raúl Núñez de Arenas Coronado
Signature · 3 lines
Linux Registered User 88736 | http://www.dervishd.net http://www.pleyades.net & http://www.gotesdelluna.net It's my PC and I'll cry if I want to... RAmen!
State of userland headers
In reply to Mariusz Mazur(I'm CCing everyone who mailed me during the past week and a half)
When I first created llh, it was a couple of weeks before 2.6.0 came out. I had a need for those headers, since a new version of PLD distro was supposed to be based on 2.6 kernels and one couldn't even build glibc with the internal headers of the last 2.5 kernels (I assume the breakage occurred a lot earlier, though).
And just because I quit, doesn't make the problem go away for my distro. Worse, currently also a couple of other projects depend on llh (LFS and uclibc are the first to come to my mind), so one way or the other, someone's eventually got to do something about it. And I did get a couple of offers to take over.
So here's how it goes.
LLH as it is is not and never will be mergable into mainline kernel. For a simple reason -- it's a piece of total crap (poetically called a hack). There are some nicely polished parts, mostly the core 'infrastructure' (endianness support, basic data types, etc.), but the rest are mostly my best guesses at 'whether this looks like a userland header', and I can't even call them educated guesses, since I simply have no clue about kernel internals. Plus count in the various bad calls I've made during those two years.
So on one side we've got a barely working hack, that is not mergable, a royal pita to maintain and gets harder to update with each kernel version. If anyone wants to take a stab at it, then be my guest, I can tell you how I've done it, but since we've got at least a couple of people directly interested in this thing, I do believe it'd be a good idea to try to do it The Right Way.
There was a thread on lkml on this topic about a year ago. IIRC I've suggested, that the best option would be to get a new set of dirs somewhere inside the kernel, and gradually export the userland usable stuff from the kernel headers, so to (a) achieve full separation and (b) avoid duplication of definitions (meaning that kernel headers would simply include the userland ones where required). Linus said, that it would break stuff and so is unacceptable.
Unfortunately I must agree with him -- I don't think it is possible to completely avoid duplication of definitions and all tries would lead to breakage of some obscure configurations -- kernel headers sometimes require various magic that should be avoided inside the userland headers at all cost. This means that initially the llh-ng project would need to start as a completely separate entity that would not require the original kernel headers for anything, and only later, after achieving some level of maturity and getting merged into the kernel, would come the time for removing some duplication. Of course that means that kernel hackers would (a) need to keep in mind that they have two places to update, (b) once llh-ng got merged, figure out where duplication can be avoided and do something about it and (c) even when that'd get done, still remember about double updates for places where duplication couldn't have been avoided.
I won't be loosing any sleep about it, though. Those mean kernel hackers never cared about us poor userland folks, so there's no reason we should feel sorry for them. Payback time.
And now for some technical details on how I see it. I'd appreciate any input.
First thing, is that we'd probably need to use a distributed rcs, since it's more flexible, and, well, distributed. Never used any, so git sounds as good an option, as anything else.
Now, what I propose we do initially, is get a bunch of headers from current llh for the most popular archs and figure out how to build all the various *libcs against them. And of course update them to the newest kernel by the way.
It'd probably be best if I did it. PLD has support for x86, x86_64, sparc, ppc and alpha, so that's what I'd take care of first. It might sound like a good idea to start with all available archs, but in reality we wouldn't even be able to test them most of the times, so it's best to stick with the archs we're directly interested in. They should be enough to get the 'design' of the thing right. And should anyone come by with a new arch he's interested in, he'd have a quick time adding it, since the foundations would already be there.
And here's where the first problem arises -- llh were so great initially, because they removed a lot of conflicts with glibc, by simply removing 'offending' linux headers and including the glibc counterparts (eg. linux/socket.h would do nothing else, than include sys/socket.h). Glibc's known for having lots of stuff simply 'hardcoded' into it's own set of headers, and more often than not, people do need to include headers from both places. I don't know about uclibc, but klibc afaik expects a lot more of the linux headers to be present, than glibc does. Hell, if llh-ng is supposed to be a full set of apis, we can't expect any of the headers to come from other places, so if any other app (libcs included) has duplicates, that's too bad for the app, since it's in need of some patching. Unfortunately the glibc's build system is one hell of a scary place to lurk around.
Ok, assuming we've got the libcs covered for the most popular archs, we're in for supporting the rest of the apps. I can just throw whatever headers we come up with at my distro and wait for the bug reports (along with colorful descriptions of how slow and painful my death is going to be), and I believe LFS guys can more or less do the same. So after some time, we'd hopefully have a bunch of mostly working headers, that are clean and rather easy to maintain (although probably require a lot of patches for various apps; but those would eventually get merged upstream; at least most of the times, I hope; keep in mind my comment about glibc being a scary and evil place). *But* in the meantime we'd be stuck with the old llh 2.6.13, so, like I've said before, if anybody's still interested in doing an update with the old llh, mail me (again) and I'll give you the details on how I've done it in the past.
And one other thing -- the assumption is that kernel<->userland apis/abis (mostly the latter) shouldn't change in an incompatible way and if they do, that's considered a bug. But until llh-ng got merged into the kernel, we'd have to figure out a way to cope with new kernel releases. Eg. what about netfilter headers, are they backwards compatible (yeah, I know iptables have their own set of netfilter headers, but apps not needing such hacks is kind of the point of the whole exercise)? And what about different paces of updates for various archs. Eg. most people are happy with the most popular archs, so it'd be a bad idea not to do a release just because we're waiting for one of the more exotic archs. How does one version such a project? Not like the old llh (x.y.z.t, where x.y.z corresponds to the kernel version), since not all releases would be always fully up to date with a given kernel. So maybe just automatic snapshots every day (assuming a change has been made). But if so, who's tree gets to be treated as the main one? (Distributed rcs, remember). And should we choose such a person, what is the chance he'll end up just like I did (a lazy schmuck that is)?
Ok, I'm getting too detailed here. One thing at a time.
So, waiting for any insights, counterproposals and/or declarations who's willing to work on old llh, llh-ng or both. Shoot away.
I'm CCing this to [email protected]
(http://lists.pld-linux.org and click away; mailman over there, so you know the drill). I've created that list about a month ago, when I figured that when people mail me, they never get to see each other's posts. I thought of deleting it a week ago, but, well, looks like it might come in handy after all.
Signature · 5 lines
Judge others by their intentions and yourself by your results.
Guy Kawasaki
Education is an admirable thing, but it is well to remember from
time to time that nothing that is worth knowing can be taught.
Oscar WildeRe: State of userland headers
Quoted 10 lines · Mariusz Mazur
This means that initially the llh-ng project would need to start as a completely separate entity that would not require the original kernel headers for anything, and only later, after achieving some level of maturity and getting merged into the kernel, would come the time for removing some duplication. Of course that means that kernel hackers would (a) need to keep in mind that they have two places to update, (b) once llh-ng got merged, figure out where duplication can be avoided and do something about it and (c) even when that'd get done, still remember about double updates for places where duplication couldn't have been avoided.
I mentioned earlier that klibc seems like a good test case. Make userspace headers that klibc can build against, and work up from there...
Quoted 16 lines
I won't be loosing any sleep about it, though. Those mean kernel hackers never cared about us poor userland folks, so there's no reason we should feel sorry for them. Payback time.
And now for some technical details on how I see it. I'd appreciate any input.
First thing, is that we'd probably need to use a distributed rcs, since it's more flexible, and, well, distributed. Never used any, so git sounds as good an option, as anything else.
Now, what I propose we do initially, is get a bunch of headers from current llh for the most popular archs and figure out how to build all the various *libcs against them. And of course update them to the newest kernel by the way.
Quoted 5 lines
It'd probably be best if I did it. PLD has support for x86,
x86_64, sparc, ppc and alpha, so that's what I'd take care of first. It might sound like a good idea to start with all available archs, but in reality we wouldn't even be able to test them most of the times, so it's best to stick with the archs we're directly interested in.
Here at timesys I've got access to arm, mips, and ppc. (Can't say I really know what I'm doing on any of them, but I'm learning.)
Quoted 9 lines
just because we're waiting for one of the more exotic archs. How does one version such a project? Not like the old llh (x.y.z.t, where x.y.z corresponds to the kernel version), since not all releases would be always fully up to date with a given kernel. So maybe just automatic snapshots every day (assuming a change has been made). But if so, who's tree gets to be treated as the main one? (Distributed rcs, remember). And should we choose such a person, what is the chance he'll end up just like I did (a lazy schmuck that is)?
Quoted 4 lines
Ok, I'm getting too detailed here. One thing at a time.
So, waiting for any insights, counterproposals and/or declarations who's willing to work on old llh, llh-ng or both. Shoot away.
I'm a little confused about the goals of the project.
In order to take advantage of new kernel features, like major/minor numbers greater than 8 bits or DMA for cd burning, we need new kernel headers. That sort of thing is why we didn't just build against 2.4 kernel headers, which for some reason seemed to be usable from userspace just fine.
You also don't want to run a libc built with newer headers than the kernel you're running on, or it'll try to use stuff that isn't there.
You're saying that the new kernel headers wouldn't be versioned using the kernel's release numbers. How do we know what kernel version their feature set matches then? (I'm confused. This happens easily...)
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: State of userland headers
Quoted 6 lines · Rob Landley
You also don't want to run a libc built with newer headers than the kernel you're running on, or it'll try to use stuff that isn't there.
You're saying that the new kernel headers wouldn't be versioned using the kernel's release numbers. How do we know what kernel version their feature set matches then? (I'm confused. This happens easily...)
That's a tradeoff. You either version the headers just like I did, meaning that a given version corresponds to a given kernel, but that means you can't release before all of the archs are fully updated (and not relying on a single person to do all of the updates is one of the points of the exercise; and with more people, one can have delays) or you're forced to figure out some other way to version the headers.
Signature · 5 lines
Judge others by their intentions and yourself by your results.
Guy Kawasaki
Education is an admirable thing, but it is well to remember from
time to time that nothing that is worth knowing can be taught.
Oscar WildeRe: State of userland headers
Quoted 8 lines · Mariusz Mazur
Quoted 6 lines · Rob Landley
You also don't want to run a libc built with newer headers than the kernel you're running on, or it'll try to use stuff that isn't there.
You're saying that the new kernel headers wouldn't be versioned using the kernel's release numbers. How do we know what kernel version their feature set matches then? (I'm confused. This happens easily...)
Quoted 6 lines
That's a tradeoff. You either version the headers just like I did, meaning that a given version corresponds to a given kernel, but that means you can't release before all of the archs are fully updated (and not relying on a single person to do all of the updates is one of the points of the exercise; and with more people, one can have delays) or you're forced to figure out some other way to version the headers.
If somebody #includes <linux/version.h> from the new headers, and can determine what kernel version it goes with for their platform from that (even if the different platforms are out of sync), and said information also goes in a README or release notes or some such, I'll live.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: State of userland headers
In reply to Mariusz MazurQuoted 1 line · Mariusz Mazur
There was a thread on lkml on this topic about a year ago. IIRC I've suggested, that the best option would be to get a new set of dirs somewhere inside the kernel, and gradually export the userland usable stuff from the kernel headers, so to (a) achieve full separation and (b) avoid duplication of definitions (meaning that kernel headers would simply include the userland ones where required). Linus said, that it would break stuff and so is unacceptable.
I seem to remember Linus saying that "breaking things is unacceptable", not that the project was guaranteed to break things (we would just need to be much more careful about it than most kernel patches). What that seems to indicate to me is that an in-kernel version would need to do the following for userspace-accessible header files for a large number of kernel releases:
#ifndef _LINUX_HEADER_H
#define _LINUX_HEADER_H
#include <kabi/header.h>
/* Define or typedef a bunch of __kabi_ prefixes to the old
prefixes they used to have in the kernel header */ #ifndef __KERNEL__
# warning "The header file <linux/header.h> is deprecated for" # warning "userspace, please use <kabi/header.h> instead." #else /* Kernel-only declarations/definitions */ #endif
If this were done carefully, all programs that compile against kernel headers could be _fixed_ in the short term (they'd go from throwing errors to giving a couple deprecation warnings). In the long term, the extra ifdeffery could be removed and the <linux/*.h> headers for which a <kabi/*.h> replacement had existed for a couple versions could be removed. New ABIs (including IOCTLs, new syscalls, etc) could be required to use <kabi/*.h> in the first place.
Quoted 1 line
Unfortunately I must agree with him -- I don't think it is possible to completely avoid duplication of definitions and all tries would lead to breakage of some obscure configurations -- kernel headers sometimes require various magic that should be avoided inside the userland headers at all cost. This means that initially the llh-ng project would need to start as a completely separate entity that would not require the original kernel headers for anything, and only later, after achieving some level of maturity and getting merged into the kernel, would come the time for removing some duplication.
I think that requiring any kind of duplication of effort on that large a scale is virtually guaranteed not to work. It will break down and be really painful for a long time.
Quoted 1 line
And here's where the first problem arises -- llh were so great initially, because they removed a lot of conflicts with glibc, by simply removing 'offending' linux headers and including the glibc counterparts (eg.
linux/socket.hwould do nothing else, than includesys/socket.h). Glibc's known for having lots of stuff simply 'hardcoded' into it's own set of headers, and more often than not, people do need to include headers from both places.
1: Ewww, bad glibc!
2: The symbols in kabi/*.h should probably all start with __kabi_
The kernel _internally_ would "#include <kabi/foo.h>" and then promptly redefine or typedef all of those objects. If the kernel or GLIBC want a struct renamed, they should #define __kabi_name_of_struct name_of_struct just before including the header file. We should also kind of frown upon non-libc userspace including many <kabi/*.h> files, since if they get included before libc can redefine the names to what it wants.
Quoted 1 line
I don't know about uclibc, but klibc afaik expects a lot more of the linux headers to be present, than glibc does.
Perhaps we should use klibc as our test-case for functional headers, then, instead of glibc.
Quoted 1 line
Hell, if llh-ng is supposed to be a full set of apis, we can't expect any of the headers to come from other places, so if any other app (libcs included) has duplicates, that's too bad for the app, since it's in need of some patching.
The solution to that is to make sure that the new exported kernel ABIs always have the __kabi_ prefix, so that we don't smash the namespace of various programs.
Cheers,
Kyle Moffett
Re: State of userland headers
Quoted 10 lines · Kyle Moffett
Quoted 8 lines · Mariusz Mazur
There was a thread on lkml on this topic about a year ago. IIRC I've suggested, that the best option would be to get a new set of dirs somewhere inside the kernel, and gradually export the userland usable stuff from the kernel headers, so to (a) achieve full separation and (b) avoid duplication of definitions (meaning that kernel headers would simply include the userland ones where required). Linus said, that it would break stuff and so is unacceptable.
Quoted 4 lines
I seem to remember Linus saying that "breaking things is unacceptable", not that the project was guaranteed to break things (we would just need to be much more careful about it than most kernel patches).
The gentoo guys clean up their own headers, apparently. I'm told they do so by moving around the #ifdef __KERNEL__ stuff to be in the correct places, and that they're currently working on a 2.6.14 or 2.6.15 version of the headers:
http://www.gentoo.org/cgi-bin/viewcvs.cgi/src/patchsets/gentoo-headers/?root=gentoo
Quoted 6 lines
What that seems to indicate to me is that an in-kernel version would need to do the following for userspace-accessible header files for a large number of kernel releases:
#ifndef
_LINUX_HEADER_H
#define_LINUX_HEADER_H
Quoted 9 lines
#include <kabi/header.h> /* Define or typedef a bunch of __kabi_ prefixes to the old prefixes they used to have in the kernel header */ #ifndef __KERNEL__ # warning "The header file <linux/header.h> is deprecated for" # warning "userspace, please use <kabi/header.h> instead." #else /* Kernel-only declarations/definitions */ #endif
Changing the #include paths in all deployed software will basically never happen. If this header package requires that, I'm not interested in it because I can't build existing software against it, and I don't expect anyone else to be.
I was thinking of possibly a parallel header set under linux-2.6.x/usr/include which the linux-2.6.x/include/*.h could #include to clean out their #ifndef __KERNEL__ stuff, and that eventually the usr/include stuff would contain approximately what Mazur's headers had contained. Unfortunately, I'm under the impression that's not a realistic approach.
Quoted 7 lines
If this were done carefully, all programs that compile against kernel headers could be _fixed_ in the short term (they'd go from throwing errors to giving a couple deprecation warnings). In the long term, the extra ifdeffery could be removed and the <linux/*.h> headers for which a <kabi/*.h> replacement had existed for a couple versions could be removed. New ABIs (including IOCTLs, new syscalls, etc) could be required to use <kabi/*.h> in the first place.
A program that includes kabi/* instead of linux/* won't build against older C libraries with older headers from older kernel versions (or older project's like Mazur's headers).
Quoted 2 lines
1: Ewww, bad glibc!
2: The symbols in kabi/*.h should probably all start with__kabi_
Any grand new incompatible thing is something I will happily ignore for as long as I am able to, and I'm not alone here. Your uptake will be zero.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: State of userland headers
Quoted 8 lines · Nix
On 24 Mar 2006, Rob Landley suggested tentatively:
Quoted 5 lines · Kyle Moffett
Quoted 2 lines
1: Ewww, bad glibc!
2: The symbols in kabi/*.h should probably all start with__kabi_Any grand new incompatible thing is something I will happily ignore for as long as I am able to, and I'm not alone here. Your uptake will be zero.
I concur. The purpose of this thing is by definition to provide libcs with the kernel/user interface stuff they need in order for userspace programs to be compiled. There's no point defining a new interface because there is a massive quantity of *existing* code out there that we must work with. (Plus, it can be, uh, difficult to get changes of this nature into glibc in particular, and glibc is the 300-pound gorilla in this particular room. If the headers don't have working with it as a goal, they are pointless.)
Hmm, I didn't really explain my idea very well. Let me start with a list of a facts. If anybody disagrees with any part of this, please let me know.
1) The <linux/*.h> headers include a lot of information essential to compiling userspace applications and libraries (libcs in particular). That same information is also required while building the kernel (IE: The ABI). 2) Those headers have a lot of declarations and definitions which must *not* be present while compiling userspace applications, and is basically kernel-only stuff. 3) Glibc is extremely large and complex 500-pound gorilla and contains an ugly build process and a lot of static definitions in its own header files that conflict with the definitions in the kernel headers. 4) UML runs into a lot of problems when glibc's headers and the native kernel headers headers conflict.
Here's some of my opinions about this:
1) Trying to create and maintain 2 separate versions of an ABI as large and complex as the kernel<=>userspace ABI across new versions and features would be extremely difficult and result in subtle bugs and missing features, even over a short period of time. 2) Ideally there should be three distinct pieces, the kernel, the ABI, and userspace. Compiling either the kernel or userspace requires the ABI, but the ABI depends only on the compiler. 3) Breaking any compatibility is bad 4) Trying to continue to maintain the glibc custom-header-file status-quo as more APIs and architectures get added to the kernel is going to become an increasingly difficult and tedious task.
My proposal (which I'm working on sample patches for) would be to divide up the kernel headers into 2 parts. The first part would be <kabi/*.h>, and the second would be all the traditional kernel-only headers. The kabi headers would *only* define things that begin with the prefix __kabi_. This would mean that the kabi headers have no risk of namespace contamination with anything else existing in the kernel or userspace, and since they would depend only on the compiler, they would be useable anywhere.
The second step would be to convert the traditional linux header to include the corresponding kabi header, then redefine its own structures and defines in terms of those in the kabi header. This would provide complete backwards compatibility to all kernel code, as well as to anything that currently compiles using the existing kernel headers. The entire rest of the <linux/*.h> header file would be wrapped in #ifdef __KERNEL__, as it should not be needed by anything in userspace.
In the process of those two steps, we would relocate many of the misplaced "#ifdef __KERNEL__" and "#endif /* __KERNEL__ */". The kabi headers should not mention __KERNEL__ at all, and the linux/* headers should be almost completely wrapped in __KERNEL__ ifdefs. That should be enough to make klibc build correctly, although from the description glibc needs significantly more work.
Once a significant portion of the kernel headers have been split that way (preserving complete backwards compatibility), external projects _may_ be converted to #include <kabi/*.h> instead of #include <linux/*.h>, although this would require other changes to the source to handle the __kabi_ prefix. Most of those should be straightforward, however. Since the kabi/*.h headers would not be kernel-version-specific, they could be copied to a system running an older kernel and reused there without problems. Even though some of the syscalls and ioctls referenced in the kabi headers might not be present on the running kernel, portable programs are expected to be able to sanely handle older kernels.
Once the kabi headers are available, it would be possible to begin cleaning up many of the glibc headers without worrying about differences between architectures. If all critical constants and datatypes are already defined in <kabi/*.h> with __kabi_ or __KABI_ prefixes, it should be possible to import those definitions into klibc and glibc without much effort.
UML has other issues with conflicts between the native kernel headers and the GLIBC-provided stubs. It's been mentioned on the prior threads about this topic that this sort of system would ease most of the issues that UML runs into.
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Cheers,
Kyle Moffett
Re: State of userland headers
Quoted 95 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett wrote:
Quoted 9 lines · Nix
On 24 Mar 2006, Rob Landley suggested tentatively:
Quoted 7 lines · Kyle Moffett
Quoted 2 lines
1: Ewww, bad glibc!
2: The symbols in kabi/*.h should probably all start with__kabi_Any grand new incompatible thing is something I will happily ignore for as long as I am able to, and I'm not alone here. Your uptake will be zero.
Quoted 8 lines
I concur. The purpose of this thing is by definition to provide libcs with the kernel/user interface stuff they need in order for userspace programs to be compiled. There's no point defining a new interface because there is a massive quantity of *existing* code out there that we must work with. (Plus, it can be, uh, difficult to get changes of this nature into glibc in particular, and glibc is the 300-pound gorilla in this particular room. If the headers don't have working with it as a goal, they are pointless.)
Hmm, I didn't really explain my idea very well. Let me start with a list of a facts. If anybody disagrees with any part of this, please let me know.
1) The <linux/*.h> headers include a lot of information essential to compiling userspace applications and libraries (libcs in particular). That same information is also required while building the kernel (IE: The ABI).
2) Those headers have a lot of declarations and definitions which must *not* be present while compiling userspace applications, and is basically kernel-only stuff.
3) Glibc is extremely large and complex 500-pound gorilla and contains an ugly build process and a lot of static definitions in its own header files that conflict with the definitions in the kernel headers.
4) UML runs into a lot of problems when glibc's headers and the native kernel headers headers conflict.Here's some of my opinions about this:
1) Trying to create and maintain 2 separate versions of an ABI as large and complex as the kernel<=>userspace ABI across new versions and features would be extremely difficult and result in subtle bugs and missing features, even over a short period of time.
2) Ideally there should be three distinct pieces, the kernel, the ABI, and userspace. Compiling either the kernel or userspace requires the ABI, but the ABI depends only on the compiler.
3) Breaking any compatibility is bad
4) Trying to continue to maintain the glibc custom-header-file status-quo as more APIs and architectures get added to the kernel is going to become an increasingly difficult and tedious task.My proposal (which I'm working on sample patches for) would be to divide up the kernel headers into 2 parts. The first part would be <kabi/*.h>, and the second would be all the traditional kernel-only headers. The kabi headers would *only* define things that begin with the prefix
__kabi_. This would mean that the kabi headers have no risk of namespace contamination with anything else existing in the kernel or userspace, and since they would depend only on the compiler, they would be useable anywhere.The second step would be to convert the traditional linux header to include the corresponding kabi header, then redefine its own structures and defines in terms of those in the kabi header. This would provide complete backwards compatibility to all kernel code, as well as to anything that currently compiles using the existing kernel headers. The entire rest of the <linux/*.h> header file would be wrapped in #ifdef
__KERNEL__, as it should not be needed by anything in userspace.In the process of those two steps, we would relocate many of the misplaced "#ifdef
__KERNEL__" and "#endif /*__KERNEL__*/". The kabi headers should not mention__KERNEL__at all, and the linux/* headers should be almost completely wrapped in__KERNEL__ifdefs. That should be enough to make klibc build correctly, although from the description glibc needs significantly more work.Once a significant portion of the kernel headers have been split that way (preserving complete backwards compatibility), external projects _may_ be converted to #include <kabi/*.h> instead of #include <linux/ *.h>, although this would require other changes to the source to handle the
__kabi_prefix. Most of those should be straightforward, however. Since the kabi/*.h headers would not be kernel-version- specific, they could be copied to a system running an older kernel and reused there without problems. Even though some of the syscalls and ioctls referenced in the kabi headers might not be present on the running kernel, portable programs are expected to be able to sanely handle older kernels.Once the kabi headers are available, it would be possible to begin cleaning up many of the glibc headers without worrying about differences between architectures. If all critical constants and datatypes are already defined in <kabi/*.h> with
__kabi_or__KABI_prefixes, it should be possible to import those definitions into klibc and glibc without much effort.
Quoted 7 lines
UML has other issues with conflicts between the native kernel headers and the GLIBC-provided stubs. It's been mentioned on the prior threads about this topic that this sort of system would ease most of the issues that UML runs into.
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Kyle,
Do you have (recorded) or recall any constraints or requirements on this $subject from Linus or Andrew or others?
I mean just basic big items, like "thou shalt not mix foo and bar".
I'm just looking for the basic parameters of this task.
Thanks,
---
~Randy
Re: State of userland headers
Quoted 4 lines · Randy.Dunlap
Kyle,
Do you have (recorded) or recall any constraints or requirements on this $subject from Linus or Andrew or others? I mean just basic big items, like "thou shalt not mix foo and bar".I'm just looking for the basic parameters of this task.
Well, to a large extent he's actually wisely kind of stayed out of the flamewars :-D I'm guessing he's hoping that we'll figure something acceptable out on our own and send him patches without him having to think about it much. He has said this, though:
Quoted 1 line
On the bigger question of what to do with kernel headers in general, let's just make one thing clear: the kernel headers are for the kernel, and big and painful re-organizations that don't help _existing_ user space are not going to happen.
He's also said this:
Quoted 7 lines · Alexandre Oliva
Quoted 1 line
Then maybe this is the fundamental problem. As long as the kernel doesn't recognize that an ABI is a contract, rather than an imposition, kernel developers won't care.
That's a silly analogy. Worse, it's a very flawed analogy.
If you want to use a legal analogy, the ABI is not a contract, it's a public _license_.
Quoted 7 lines
Why? A contract is something that you cannot change unilaterally. Clearly the kernel _can_ and will change the ABI without asking every existing program for permission.
In a license, you can always just _expand_ the license eventually. Maybe you originally licensed for "fly-fishing for trout", and later you can expand that license to say "you can also catch crawfish" without impacting your existing licensees.
And exactly as with an ABI, the only thing you can't do is _remove_ rights without getting into trouble.
So get your analogies straight. The kernel ABI is _not_ a contract.
My hope for this is that we can start doing _little_ and clearly obvious changes that clean up header files. Basically a lot of the __KERNEL__ definitions need janitorial work. Either they need to be removed or they need to be put in the right places. Also it seems like there's a lot of duplication between architectures; for one example look at all the varieties of FD_SET code in the different linux/include/asm-*/posix_types.h files. That's one of the areas I'm trying to clean up into a single linux/fdset.h included from the pertinent files. Notice how the file forcibly overrides GLIBC; I think if we can define it as __KABI_FD* and only define the non-prefixed version in __KERNEL__, then we can provide something that GLIBC and klibc can get for free, without having to figure out their own bitmaps. Likewise some of the user-accessible types are uniform across architectures, and others are haphazardly arrayed, some to be compatible with other OSes on that architecture, others just because that's what the arch they copied from used. I'm hoping if I can get enough small patches flowing, others will join in too and the process will go easier.
Cheers,
Kyle Moffett
Re: State of userland headers
Just to through this out, I'm working on project right now where we want to use the latest Kernel Headers, but since LLH is pretty much dead we came up with a interim solution until something else gets worked out. The script lists below only sanitizes the headers that are needed by userspace, there may be some missing, but this has worked on the systems I've built up to this point.
http://ftp.jg555.com/headers/headers2
Signature · 3 lines
---- Jim Gifford [email protected]
Re: State of userland headers
In reply to Kyle MoffettQuoted 7 lines · Kyle Moffett
4) UML runs into a lot of problems when glibc's headers and the native kernel headers headers conflict.
UML has other issues with conflicts between the native kernel headers and the GLIBC-provided stubs. It's been mentioned on the prior threads about this topic that this sort of system would ease most of the issues that UML runs into.
Actually, this isn't quite the same as what UML hits. I have an amicable solution (with some warts) to the glibc/kernel header conflicts - files build against either glibc headers or kernel headers, but never both.
The warts are where I pass information between those two sets of files that could be interpretted differently on either side, but aren't because both sides are Linux. For example, I freely pass errno values across that interface in the hope that the glibc headers and the kernel headers agree on what they mean.
My problem with the kernel headers is that they are a mixture of things that are usable in userspace and things that aren't. This is closely related, but not identical to, things which are part of the ABI and things which aren't.
For example, the kernel locks are quite usable in userspace, but you would never make them part of the ABI.
So, a set of KABI headers would likely make UML's headers cleaner, by avoiding copying arch headers and using various nasty tricks to disable objectionable pieces of headers which I steal from the arch.
So what I really want is a superset of the KABI headers, but the KABI will give me most of what I want.
Jeff
Re: State of userland headers
Quoted 6 lines · Jeff Dike
Quoted 4 lines · Kyle Moffett
4) UML runs into a lot of problems when glibc's headers and the native kernel headers headers conflict.
UML has other issues with conflicts between the native kernel headers and the GLIBC-provided stubs. It's been mentioned on the prior threads about this topic that this sort of system would ease most of the issues that UML runs into.
Quoted 7 lines
Actually, this isn't quite the same as what UML hits. My problem with the kernel headers is that they are a mixture of things that are usable in userspace and things that aren't. This is closely related, but not identical to, things which are part of the ABI and things which aren't.
For example, the kernel locks are quite usable in userspace, but you would never make them part of the ABI.
So, a set of KABI headers would likely make UML's headers cleaner, by avoiding copying arch headers and using various nasty tricks to disable objectionable pieces of headers which I steal from the arch.
So what I really want is a superset of the KABI headers, but the KABI will give me most of what I want.
So perhaps could we define an informal subset of the kernel code that works in both userspace and kernel-space and put it in include/libk? Stuff like linked lists, spinlocks (depends on arch, may not be supported), etc could be in linux/libk and linux/include/libk or similar, and then from there included into linux/include/linux/list.h, etc, as well as into the UML files that need it. Since the provider and user would both be the Linux kernel, I see no issues with trying to provide a stable interface of any kind, especially if we document it as "PRIVATE - FOR KERNEL USE ONLY!!!" with big warning signs. As a nice bonus, this would make it possible to implement some user-space unit tests of various pieces.
Cheers,
Kyle Moffett
Re: State of userland headers
Quoted 11 lines · Kyle Moffett
So perhaps could we define an informal subset of the kernel code that works in both userspace and kernel-space and put it in
include/libk? Stuff like linked lists, spinlocks (depends on arch, may not be supported), etc could be in linux/libk and linux/include/libk or similar, and then from there included into linux/include/linux/list.h, etc, as well as into the UML files that need it. Since the provider and user would both be the Linux kernel, I see no issues with trying to provide a stable interface of any kind, especially if we document it as "PRIVATE - FOR KERNEL USE ONLY!!!" with big warning signs. As a nice bonus, this would make it possible to implement some user-space unit tests of various pieces.
This would be perfect.
Jeff
Re: State of userland headers
In reply to Kyle MoffettQuoted 3 lines · Kyle Moffett
however. Since the kabi/*.h headers would not be kernel-version- specific, they could be copied to a system running an older kernel and reused there without problems.
Since when is the kernel ABI not kernel version specific? You can use an older ABI on a newer kernel, but you can't use a newer ABI on an older kernel.
Quoted 4 lines
Even though some of the syscalls
and ioctls referenced in the kabi headers might not be present on the running kernel, portable programs are expected to be able to sanely handle older kernels.
At the source level, maybe. At the binary level? Not really.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: State of userland headers
Quoted 4 lines · Rob Landley
Quoted 1 line · Kyle Moffett
however. Since the kabi/*.h headers would not be kernel-version-specific, they could be copied to a system running an older kernel and reused there without problems.
Since when is the kernel ABI not kernel version specific? You can use an older ABI on a newer kernel, but you can't use a newer ABI on an older kernel.
By the very definition of "ABI", it _must_ not be kernel version specific. A program written for a newer ABI and compiled against it _must_ be able to gracefully handle syscalls not existing when run on an older kernel. The program may fail to work entirely because the syscalls it needs are not present, but that's no different than a "Kernel does not have feature CONFIG_FOO enabled" issue.
Quoted 3 lines
Quoted 1 line
Even though some of the syscalls and ioctls referenced in the kabi headers might not be present on the running kernel, portable programs are expected to be able to sanely handle older kernels.
At the source level, maybe. At the binary level? Not really.
The ABI must be compatible at both source and binary interfaces, "Application _Binary_ Interface" is the name, after all. It's OK to introduce new features so long as you can gracefully handle them not being present. Look for example at what happens if I take V4L apps and run them on a 2.4.0 kernel without V4L. They report that the kernel is too old or missing features, but aside from V4L not being present, everything just works.
As a result, I think that if we can clean up the headers in kernel, we can reuse those same headers for compiling userspace everywhere, even if the userspace is going to be run on an older kernel.
Cheers,
Kyle Moffett
cover KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 2 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
1) The Linux kernel compiles and works at every step along the way
2) Since most of the headers are currently quite broken with respect to GLIBC and userspace, I won't spend much extra time preserving compatibility with GLIBC, userspace, or non-GCC compilers.
3) Everything in include/kabi will have a __kabi_ or __KABI_ prefix.
4) Headers in include/linux that need the KABI interfaces will include the corresponding <kabi/*.h> header and define or typedef the necessary KABI definitions to the names the kernel wants.
5) The stuff in include/kabi/*.h should always be completely independent of userspace/kernelspace and not require any includes outside of <kabi/*>. This means that the only preprocessor symbols that we can assume are present are those provided by the compiler itself.
Cheers,
Kyle Moffett
RFC PATCH 1/2 Create initial kernel ABI header infrastructure
Create initial kernel ABI header infrastructure
The model proposed for the new kernel ABI headers is that the headers pertinent to both kernelspace and userspace would be reorganized and cleaned out into include/kabi. The new headers should stay in a restricted namespace such that they are useable anywhere. Just as include/asm is symlinked to include/asm-$ARCH, include/kabi/arch will be symlinked to include/kabi/arch-$ARCH to provide a straightforward way of selecting the correct architecture ABI.
This creates the first architecture-specific header file, "stddef.h", which implements a basic BITS_PER_LONG definition, and adds the corresponding Makefile bits to create the include/kabi/arch symlink.
There's probably something else that needs to be done about getting the correct include/kabi/* files in the tarball but I'm not aware of it at this time, so I'll just pretend it doesn't exist :-D.
This appears to compile correctly with allmodconfig on powerpc (32-bit)
---
commit affadc51c2b4c96785809a352973a9f6a51a4c37
tree e0979fa864eab8902ccae459060b18f077ad1d4e
parent 36ddf5bbdea7ba4582abc62f106f0f0e9f0b6b91
author Kyle Moffett <[email protected]> Sun, 26 Mar 2006 06:18:06 -0500 committer Kyle Moffett <[email protected]> Sun, 26 Mar 2006 06:18:06 -0500
Makefile | 22 +++++++++++++++++++--- include/kabi/arch-alpha/stddef.h | 7 +++++++ include/kabi/arch-arm/stddef.h | 7 +++++++ include/kabi/arch-arm26/stddef.h | 7 +++++++ include/kabi/arch-cris/stddef.h | 7 +++++++ include/kabi/arch-frv/stddef.h | 7 +++++++ include/kabi/arch-h8300/stddef.h | 7 +++++++ include/kabi/arch-i386/stddef.h | 7 +++++++ include/kabi/arch-ia64/stddef.h | 7 +++++++ include/kabi/arch-m32r/stddef.h | 7 +++++++ include/kabi/arch-m68k/stddef.h | 7 +++++++ include/kabi/arch-m68knommu/stddef.h | 1 + include/kabi/arch-mips/stddef.h | 12 ++++++++++++ include/kabi/arch-parisc/stddef.h | 12 ++++++++++++ include/kabi/arch-powerpc/stddef.h | 12 ++++++++++++ include/kabi/arch-ppc/stddef.h | 1 + include/kabi/arch-s390/stddef.h | 12 ++++++++++++ include/kabi/arch-sh/stddef.h | 7 +++++++ include/kabi/arch-sh64/stddef.h | 7 +++++++ include/kabi/arch-sparc/stddef.h | 7 +++++++ include/kabi/arch-sparc64/stddef.h | 7 +++++++ include/kabi/arch-v850/stddef.h | 7 +++++++ include/kabi/arch-x86_64/stddef.h | 7 +++++++ include/kabi/arch-xtensa/stddef.h | 7 +++++++ 24 files changed, 188 insertions(+), 3 deletions(-)
Makefile +19 -3
@@ -787,13 +787,15 @@ ifneq ($(KBUILD_SRC),) /bin/false; \ fi; $(Q)if [ ! -d include2 ]; then mkdir -p include2; fi; + $(Q)if [ ! -d include2/kabi ]; then mkdir -p include2/kabi; fi; $(Q)ln -fsn $(srctree)/include/asm-$(ARCH) include2/asm + $(Q)ln -fsn $(srctree)/include/kabi/arch-$(ARCH) include2/kabi/arch endif # prepare2 creates a makefile if using a separate output directory prepare2: prepare3 outputmakefile -prepare1: prepare2 include/linux/version.h include/asm \ +prepare1: prepare2 include/linux/version.h include/asm include/kabi/arch \ include/config/MARKER ifneq ($(KBUILD_MODULES),) $(Q)rm -rf $(MODVERDIR)
@@ -822,6 +824,12 @@ include/asm: $(Q)if [ ! -d include ]; then mkdir -p include; fi; @ln -fsn asm-$(ARCH) $@ +include/kabi/arch: + @echo ' SYMLINK $@ -> include/kabi/arch-$(ARCH)' + $(Q)if [ ! -d include ]; then mkdir -p include; fi; + $(Q)if [ ! -d include/kabi ]; then mkdir -p include/kabi; fi; + @ln -fsn arch-$(ARCH) $@ + # Split autoconf.h into include/linux/config/* include/config/MARKER: scripts/basic/split-include include/linux/autoconf.h
@@ -944,7 +952,8 @@ CLEAN_FILES += vmlinux System.map \ MRPROPER_DIRS += include/config include2 MRPROPER_FILES += .config .config.old include/asm .version .old_version \ include/linux/autoconf.h include/linux/version.h \ - .kernelrelease Module.symvers tags TAGS cscope* + .kernelrelease Module.symvers tags TAGS cscope* \ + include/kabi/arch # clean - Delete most, but leave enough to build external modules #
@@ -1195,12 +1204,19 @@ define all-sources find $(__srctree)security/selinux/include $(RCS_FIND_IGNORE) \ -name '*.[chS]' -print; \ find $(__srctree)include $(RCS_FIND_IGNORE) \ - \( -name config -o -name 'asm-*' \) -prune \ + \( -name config -o -name 'asm-*' -o -name 'kabi' \) -prune \ + -o -name '*.[chS]' -print; \ + find $(__srctree)include/kabi $(RCS_FIND_IGNORE) \ + \( -name 'arch-*' \) -prune \ -o -name '*.[chS]' -print; \ for ARCH in $(ALLINCLUDE_ARCHS) ; do \ find $(__srctree)include/asm-$${ARCH} $(RCS_FIND_IGNORE) \ -name '*.[chS]' -print; \ done ; \ + for ARCH in $(ALLINCLUDE_ARCHS) ; do \ + find $(__srctree)include/kabi/arch-$${ARCH} $(RCS_FIND_IGNORE) \ + -name '*.[chS]' -print; \ + done ; \ find $(__srctree)include/asm-generic $(RCS_FIND_IGNORE) \ -name '*.[chS]' -print ) endef
include/kabi/arch-alpha/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_ALPHA_STDDEF_H +# define __KABI_ARCH_ALPHA_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 + +#endif /* not __KABI_ARCH_ALPHA_STDDEF_H */
include/kabi/arch-arm/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_ARM_STDDEF_H +# define __KABI_ARCH_ARM_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_ARM_STDDEF_H */
include/kabi/arch-arm26/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_ARM26_STDDEF_H +# define __KABI_ARCH_ARM26_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_ARM26_STDDEF_H */
include/kabi/arch-cris/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_CRIS_STDDEF_H +# define __KABI_ARCH_CRIS_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_CRIS_STDDEF_H */
include/kabi/arch-frv/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_FRV_STDDEF_H +# define __KABI_ARCH_FRV_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_FRV_STDDEF_H */
include/kabi/arch-h8300/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_H8300_STDDEF_H +# define __KABI_ARCH_H8300_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_H8300_STDDEF_H */
include/kabi/arch-i386/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_I386_STDDEF_H +# define __KABI_ARCH_I386_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_I386_STDDEF_H */
include/kabi/arch-ia64/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_IA64_STDDEF_H +# define __KABI_ARCH_IA64_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 + +#endif /* not __KABI_ARCH_IA64_STDDEF_H */
include/kabi/arch-m32r/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_M32R_STDDEF_H +# define __KABI_ARCH_M32R_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_M32R_STDDEF_H */
include/kabi/arch-m68k/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_M68K_STDDEF_H +# define __KABI_ARCH_M68K_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_M68K_STDDEF_H */
include/kabi/arch-m68knommu/stddef.h new file +1 -0
@@ -0,0 +1 @@ +#include <kabi/arch-m68k/stddef.h>
include/kabi/arch-mips/stddef.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef __KABI_ARCH_MIPS_STDDEF_H +# define __KABI_ARCH_MIPS_STDDEF_H 1 + +# if _MIPS_SZLONG ==64 +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 +# else +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 +# endif + +#endif /* not __KABI_ARCH_MIPS_STDDEF_H */
include/kabi/arch-parisc/stddef.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef __KABI_ARCH_PARISC_STDDEF_H +# define __KABI_ARCH_PARISC_STDDEF_H 1 + +# ifdef __LP64__ +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 +# else +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 +# endif + +#endif /* not __KABI_ARCH_PARISC_STDDEF_H */
include/kabi/arch-powerpc/stddef.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef __KABI_ARCH_POWERPC_STDDEF_H +# define __KABI_ARCH_POWERPC_STDDEF_H 1 + +# ifdef __powerpc64__ +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 +# else +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 +# endif + +#endif /* not __KABI_ARCH_POWERPC_STDDEF_H */
include/kabi/arch-ppc/stddef.h new file +1 -0
@@ -0,0 +1 @@ +#include <kabi/arch-powerpc/stddef.h>
include/kabi/arch-s390/stddef.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef __KABI_ARCH_S390_STDDEF_H +# define __KABI_ARCH_S390_STDDEF_H 1 + +# ifdef __s390x__ +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 +# else +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 +# endif + +#endif /* not __KABI_ARCH_S390_STDDEF_H */
include/kabi/arch-sh/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_SH_STDDEF_H +# define __KABI_ARCH_SH_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_SH_STDDEF_H */
include/kabi/arch-sh64/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_SH64_STDDEF_H +# define __KABI_ARCH_SH64_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_SH64_STDDEF_H */
include/kabi/arch-sparc/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_SPARC_STDDEF_H +# define __KABI_ARCH_SPARC_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_SPARC_STDDEF_H */
include/kabi/arch-sparc64/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_SPARC64_STDDEF_H +# define __KABI_ARCH_SPARC64_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 + +#endif /* not __KABI_ARCH_SPARC64_STDDEF_H */
include/kabi/arch-v850/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_V850_STDDEF_H +# define __KABI_ARCH_V850_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_V850_STDDEF_H */
include/kabi/arch-x86_64/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_X86_64_STDDEF_H +# define __KABI_ARCH_X86_64_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 64 +# define __KABI_BITS_PER_LONG_SHIFT 6 + +#endif /* not __KABI_ARCH_X86_64_STDDEF_H */
include/kabi/arch-xtensa/stddef.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __KABI_ARCH_XTENSA_STDDEF_H +# define __KABI_ARCH_XTENSA_STDDEF_H 1 + +# define __KABI_BITS_PER_LONG 32 +# define __KABI_BITS_PER_LONG_SHIFT 5 + +#endif /* not __KABI_ARCH_XTENSA_STDDEF_H */
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 1 line · Kyle Moffett
Create initial kernel ABI header infrastructure
it's nice that you picked this one;
for this you want an arch-generic/stddef32.h and stddef64.h
and have arch-foo just only include the proper generic one..
(and... why do you prefix these with _KABI? that's a mistake imo. Don't both with that. Really. Either these need exporting to userspace, but then either use __ as prefix or don't use a prefix. But KABI.. No.)
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 7 lines · Arjan van de Ven
Quoted 1 line · Kyle Moffett
Create initial kernel ABI header infrastructure
it's nice that you picked this one;
for this you want anarch-generic/stddef32.handstddef64.hand have arch-foo just only include the proper generic one..
I plan to add a lot of other definitions to this file later on. For example different architectures have different notions of what a __kernel_ino_t is (unsigned int versus unsigned long). I may rename this file as types.h, but from looking through the code I figure I'll have enough general purpose declarations about "This architecture has blah" that a separate stddef.h file will be worth it.
Quoted 1 line
(and... why do you prefix these with
_KABI? that's a mistake imo. Don't bother with that. Really. Either these need exporting to userspace, but then either use __ as prefix or don't use a prefix. But KABI.. No.)
According to the various standards all symbols beginning with __ are reserved for "The Implementation", including the compiler, the standard library, the kernel, etc. In order to avoid clashing with any/all of those, I picked the __KABI_ and __kabi_ prefixes for uniqueness. In theory I could just use __, but there are problems with that too. For example, note how the current compiler.h files redefine __always_inline to mean something kinda different. The GCC manual says we should be able to write this:
inline __attribute__((__always_inline)) int increment(int x)
{
return x+1;
}Except when compiling the kernel headers turn that into this (which obviously doesn't compile):
inline __attribute__((__attribute__((always_inline)))) int increment(int x)
{
return x+1;
}As a result, I kinda want to stay away from anything that remotely looks like a conflicting namespace. Using such a unique namespace means we can also safely do this if necessary (Since you can't "typedef struct foo struct bar"):
kabi/foo.h:
struct __kabi_foo {
int x;
int y;
};
linux/foo.h:
#define __kabi_foo foo
#include <kabi/foo.h>
drivers/foo/foo.h:
#include <linux/foo.h>
void func()
{
struct foo = { .x = 1, .y = 2 };
}
Cheers,
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Hello!
Quoted 11 lines
As a result, I kinda want to stay away from anything that remotely looks like a conflicting namespace. Using such a unique namespace means we can also safely do this if necessary (Since you can't "typedef struct foo struct bar"):
kabi/foo.h:
struct__kabi_foo{
int x;
int y;
};
Quoted 3 lines
linux/foo.h:
#define__kabi_foofoo
#include <kabi/foo.h>
This looks very fragile -- <kabi/foo.h> can be included earlier by another header.
Have a nice fortnight
Signature · 3 lines
Martin `MJ' Mares <[email protected]> http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth And God said: E = 1/2mv^2 - Ze^2/r ...and there *WAS* light!
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 10 lines · Martin Mares
Hello!
Quoted 8 lines
As a result, I kinda want to stay away from anything that remotely looks like a conflicting namespace. Using such a unique namespace means we can also safely do this if necessary (Since you can't "typedef struct foo struct bar"):
kabi/foo.h:
struct__kabi_foo{
int x;
int y;
};
Quoted 5 lines
Quoted 3 lines
linux/foo.h:
#define__kabi_foofoo
#include <kabi/foo.h>This looks very fragile -- <
kabi/foo.h> can be included earlier by another header.
It _is_ fragile, but for a number of POSIX-defined structs that's actually the only way to do it without duplicating the data structure in entirety, unless the GCC people can implement a "typedef struct foo struct bar;" Hopefully it would be a relatively rare occurrence and carefully thought out in any case. We may end up with some duplication regardless :-\.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Hello!
Quoted 4 lines
It _is_ fragile, but for a number of POSIX-defined structs that's actually the only way to do it without duplicating the data structure in entirety, unless the GCC people can implement a "typedef struct foo struct bar;"
Actually, something like that can be achieved using anonymous structure members:
struct xxx {
struct yyy;
};However, I'm not sure how old versions of GCC support this feature.
Have a nice fortnight
Signature · 3 lines
Martin `MJ' Mares <[email protected]> http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth And God said: E = 1/2mv^2 - Ze^2/r ...and there *WAS* light!
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 7 lines · Martin Mares
Quoted 1 line
It _is_ fragile, but for a number of POSIX-defined structs that's actually the only way to do it without duplicating the data structure in entirety, unless the GCC people can implement a "typedef struct foo struct bar;"
Actually, something like that can be achieved using anonymous structure members:
struct xxx { struct yyy; };
Oh, if only that worked. Actually, what happens is the "struct yyy;" declaration inside of struct xxx looks just like "struct yyy;" out in the middle of some random header file. It predeclares the existence of a struct yyy and does nothing else.
For instance, the following sample program:
struct foo {
int a;
int b;
};
struct bar {
struct foo;
};
int main()
{
struct foo myfoo = { .a = 1, .b = 2 };
struct bar mybar = { .a = 1, .b = 2 };
return 0;
}Compiled like this:
gcc mytest.c -o mytest
Generates these errors:
mytest.c:7: warning: declaration does not declare anything mytest.c: In function `main':
mytest.c:12: error: unknown field a' specified in initializer mytest.c:12: warning: excess elements in struct initializer mytest.c:12: warning: (near initialization for mybar') mytest.c:12: error: unknown field b' specified in initializer mytest.c:12: warning: excess elements in struct initializer mytest.c:12: warning: (near initialization for mybar')
Cheers,
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
In reply to Kyle MoffettQuoted 9 lines
According to the various standards all symbols beginning with __ are reserved for "The Implementation", including the compiler, the standard library, the kernel, etc. In order to avoid clashing with any/all of those, I picked the
__KABI_and__kabi_prefixes for uniqueness. In theory I could just use __, but there are problems with that too. For example, note how the currentcompiler.hfiles redefine__always_inlineto mean something kinda different.
well... the "problem" is there today, and... it's not much of a problem if at all; there's just a few simple rules to keep it that way which seem to wkr.
And your __alway_inline example.. that's something that really is kernel internal and shouldn't be exposed to userland.
I think the "problem" really is not there if
1) we only use __ symbols like we do today for non-structs
2) avoid including kernel headers in kernel headers as far as possible. This means, that if an application wants to use MTD struct "struct mtd_foo" it will have to include the MTD header, but that he otherwise never gets it. Eg all such symbols are in a "Yes I really want it" header.
3) keep the userspace exposed stuff as small as reasonable. Your __always_inline example doesn't make that cut. A struct used as ioctl of a subsystem/driver in the header specially for that subsystem/driver does.
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 3 lines · Arjan van de Ven
Quoted 1 line
According to the various standards all symbols beginning with __ are reserved for "The Implementation", including the compiler, the standard library, the kernel, etc. In order to avoid clashing with any/all of those, I picked the
__KABI_and__kabi_prefixes for uniqueness. In theory I could just use __, but there are problems with that too. For example, note how the currentcompiler.hfiles redefine__always_inlineto mean something kinda different.well... the "problem" is there today, and... it's not much of a problem if at all; there's just a few simple rules to keep it that way which seem to work.
The current rules work for things kept completely private to the kernel. If we start exporting things into the guts of libc, we can expect that those guys have _also_ been using all sorts of double-underscore symbols in conflicting ways.
Quoted 1 line
And your
__alway_inlineexample.. that's something that really is kernel internal and shouldn't be exposed to userland.
Yes, but it illustrates how a supposedly-innocuous double-underscore symbol actually conflicts with a GCC builtin, and that's just GCC! That's not including the massive "The Implementation" codebase that is GLIBC.
Quoted 1 line
2) avoid including kernel headers in kernel headers as far as possible. This means, that if an application wants to use MTD struct "struct
mtd_foo" it will have to include the MTD header, but that he otherwise never gets it. Eg all such symbols are in a "Yes I really want it" header.
This is Hard(TM). Take a look at the interdependent mess which is kernel.h, or sched.h. Feel free to submit patches :-D That's actually part of the reason why I'm trying this kabi/*.h bit. I think that if I can pull out userspace-required struct definitions from the rest of the code, we'll notice that some of the header files don't need as many dependencies anymore and can be cleaned out a bit.
I've also been looking at GCC's "precompiled header" stuff. I think that a single kabi.h file that includes all other kabi headers, precompiled to "kabi.h.gch", could potentially speed up a portion of the linux kernel build by removing a lot of text parsing. I can't tell for sure till I've cleaned up more crap and done benchmarks, but I'm willing to give it a shot. :-D
Cheers,
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
In reply to Kyle MoffettHi!
Quoted 14 lines
I plan to add a lot of other definitions to this file later on. For example different architectures have different notions of what a
__kernel_ino_tis (unsigned int versus unsigned long). I may rename this file astypes.h, but from looking through the code I figure I'll have enough general purpose declarations about "This architecture has blah" that a separatestddef.hfile will be worth it.Quoted 4 lines
(and... why do you prefix these with
_KABI? that's a mistake imo. Don't bother with that. Really. Either these need exporting to userspace, but then either use __ as prefix or don't use a prefix. But KABI.. No.)
Quoted 10 lines
According to the various standards all symbols beginning with __ are reserved for "The Implementation", including the compiler, the standard library, the kernel, etc. In order to avoid clashing with any/all of those, I picked the
__KABI_and__kabi_prefixes for uniqueness. In theory I could just use __, but there are problems with that too. For example, note how the currentcompiler.hfiles redefine__always_inlineto mean something kinda different. The GCC manual says we should be able to write this:
__KABI_ everywhere will just make your headers totally unreadable. Please don't do that.
Signature · 1 line
Thanks, Sharp!
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 27 lines · On Wed, 29 Mar 2006 22:26:41 +0000 Pavel Machek wrote:
Hi!
Quoted 14 lines
I plan to add a lot of other definitions to this file later on. For example different architectures have different notions of what a
__kernel_ino_tis (unsigned int versus unsigned long). I may rename this file astypes.h, but from looking through the code I figure I'll have enough general purpose declarations about "This architecture has blah" that a separatestddef.hfile will be worth it.Quoted 4 lines
(and... why do you prefix these with
_KABI? that's a mistake imo. Don't bother with that. Really. Either these need exporting to userspace, but then either use __ as prefix or don't use a prefix. But KABI.. No.)Quoted 10 lines
According to the various standards all symbols beginning with __ are reserved for "The Implementation", including the compiler, the standard library, the kernel, etc. In order to avoid clashing with any/all of those, I picked the
__KABI_and__kabi_prefixes for uniqueness. In theory I could just use __, but there are problems with that too. For example, note how the currentcompiler.hfiles redefine__always_inlineto mean something kinda different. The GCC manual says we should be able to write this:
Quoted 2 lines
__KABI_everywhere will just make your headers totally unreadable. Please don't do that.
Ack, I agree.
---
~Randy
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 11 lines · Randy.Dunlap
Quoted 8 lines · On Wed, 29 Mar 2006 22:26:41 +0000 Pavel Machek wrote:
Quoted 6 lines
I plan to add a lot of other definitions to this file later on. For example different architectures have
different notions of what a__kernel_ino_tis (unsigned int versus unsigned long). I may rename this file astypes.h, but from looking through the code I figure I'll have enough general purpose declarations about "This architecture has blah" that a separatestddef.hfile will be worth it.Quoted 1 line
(and... why do you prefix these with
_KABI? that's a mistake imo. Don't bother with that. Really. Either these need exporting to userspace, but then either use __ as prefix or don't use a prefix. But KABI.. No.)According to the various standards all symbols beginning with __ are reserved for "The Implementation", including the compiler, the standard library, the kernel, etc. In order to avoid clashing with any/all of those, I picked the
__KABI_and__kabi_prefixes for uniqueness. In theory I could just use __, but there are problems with that too. For example, note how the currentcompiler.hfiles redefine__always_inlineto mean something kinda different. The GCC manual says we should be able to write this:
__KABI_everywhere will just make your headers totally unreadable. Please don't do that.Ack, I agree.
Let me reiterate two facts:
(1) The various C standards state that the implementation should restrict itself to symbols prefixed with "__", everything else is reserved for user code (Including symbols prefixed with a single underscore).
(2) GCC predefines a large collection of symbols, macros, and functions for its own use, and this set is not constant (just look at the number of new __-prefixed symbols added between GCC 3 and 4. In addition, we're not just compiling this code under GCC, but people will also be using it (hopefully unmodified) under tiny-cc, intel-cc, PGI, PathScale, Lahey, ARM Ltd, lcc, and possibly others. It probably does not need to be stated that for something as userspace-sensitive as the KABI headers we should not risk colliding with predefined builtins in any of those compilers.
So my question to the list is this:
Can you come up with any way other than using a "__kabi_" prefix to reasonably avoid namespace collisions with that large list of compilers? If you have some way, I'd be interested to hear it, but as a number of those compilers are commercial I'd have no way to test on them (and I suspect most people on this list would not either).
Of course, if the general consensus is that supporting non-GCC is not important, then that's ok with me. Judging from the number of negative responses my earlier "[OT] Non-GCC compilers used for linux userspace" got, however, that doesn't seem to be the case.
Thanks for the advice!
Cheers,
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 4 lines · Kyle Moffett
(1) The various C standards state that the implementation should restrict itself to symbols prefixed with "__", everything else is reserved for user code (Including symbols prefixed with a single underscore).
user code in this context includes, in my interpretation, headers for specific ioctl structures and such (eg direct included headers; headers that those headers include are a different matter) that the user wants.
Which is the vast majority of the kernel abi.
Exceptions are things like __u64 which are almost always "indirectly" included. Eg the app wants "struct foo_ioctl" and foo_ioctl happens to contain an __u64 which needs types.h
Quoted 9 lines
(2) GCC predefines a large collection of symbols, macros, and functions for its own use, and this set is not constant (just look at the number of new __-prefixed symbols added between GCC 3 and 4. In addition, we're not just compiling this code under GCC, but people will also be using it (hopefully unmodified) under tiny-cc, intel-cc, PGI, PathScale, Lahey, ARM Ltd, lcc, and possibly others. It probably does not need to be stated that for something as userspace- sensitive as the KABI headers we should not risk colliding with predefined builtins in any of those compilers.
the good news is that as kernel we have SOME power ;) people who make compilers like that stay away from symbols the kernel defines. Especially gcc is very careful in its namespace use and tends to be very different from what linux kernel uses
Quoted 4 lines
So my question to the list is this:
Can you come up with any way other than using a "__kabi_" prefix to reasonably avoid namespace collisions with that large list of compilers?
First of all be realistic. Don't do silly things for places where it doesn't matter. Again the ioctl structs come to mind. Second, names the kernel ALREADY claims are of course free to use as well; all those compilers ALREADY stay away from those.
What is left if you take those two big ones out of the picture?
Quoted 4 lines
Of course, if the general consensus is that supporting non-GCC is not important, then that's ok with me. Judging from the number of negative responses my earlier "[OT] Non-GCC compilers used for linux userspace" got, however, that doesn't seem to be the case.
there is another dynamic; those other compilers in general try to emulate gcc to a large degree, usually a very large degree. So the "problem" you see is a lot smaller than you think.
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 7 lines · Arjan van de Ven
Quoted 1 line · Kyle Moffett
(1) The various C standards state that the implementation should restrict itself to symbols prefixed with "__", everything else is reserved for user code (Including symbols prefixed with a single underscore).
user code in this context includes, in my interpretation, headers for specific ioctl structures and such (eg direct included headers; headers that those headers include are a different matter) that the user wants.
Which is the vast majority of the kernel abi. Exceptions are things like
__u64which are almost always "indirectly" included. Eg the app wants "structfoo_ioctl" andfoo_ioctlhappens to contain an__u64which needstypes.h
I get the impression that you're confusing my KABI proposal with kernel-only code. The point of KABI is to generalize some of the headers so that they work both in userspace and kernelspace, even if it means that the headers themselves aren't as clean as they might otherwise be. In the kernel we have only one API where we have to deal with compatibility: the kernel/userspace boundary. I'd like to strictly define all of that compatibility within a single set of _upstream_ headers in linux/include/linux-kabi or similar.
But returning to your statements above, user code in that (C standards) context means anything not standard-specified. Basically, from userspace, a program that includes _any_ standard-specified header must bring in *only* those symbols specified by the standard as being present in that header, as well as any implementation-defined double-underscore symbols. So unless I "#include <stdint.h>", my user program can typedef uint32_t to anything it wants, even a struct, if it likes. I've actually done that before on a very strange Cray platform with non-GCC compiler and OS that only supported 64-bit integers (a char was a 64-bit integer with ops to mask out the upper 56 bits), just to make bzip2 work as expected. The same rules apply for _every_ other standard-specified header (stdlib.h, stdio.h, sys/types.h, etc).
Quoted 1 line
the good news is that as kernel we have SOME power ;) people who make compilers like that stay away from symbols the kernel defines. Especially gcc is very careful in its namespace use and tends to be very different from what linux kernel uses.
As I said above, I'm talking about the proposed kernel ABI headers here. The whole point of these is something that both the kernel _and_ userspace can include. This means that the headers must allow a libc that includes them to be completely standards-compliant. It also means that these headers must work not only for GCC, but also for tcc and icc and the myriad of other compilers I mentioned (the ones that I got so many emails about in the "Non-GCC compilers" thread).
Quoted 3 lines
Quoted 1 line
So my question to the list is this: Can you come up with any way other than using a "
__kabi_" prefix to reasonably avoid namespace collisions with that large list of compilers?First of all be realistic. Don't do silly things for places where it doesn't matter. Again the ioctl structs come to mind. Second, names the kernel ALREADY claims are of course free to use as well; all those compilers ALREADY stay away from those.
Agreed (to some extent). I'm still evaluating a lot of the code, I plan to submit several non-KABI cleanups first to prepare the way. I'll try to get you specific patches so you can give me specific rejections, ok? :-D I have the feeling that we both see things the same way but have difficulty communicating due to the lack of examples.
Quoted 1 line
What is left if you take those two big ones out of the picture?
First of all, there seems to be a lot of little bits of code that the KABI headers really want to use but can't right now. For example, the asm-*/bitops.h headers contain inline functions for a variety of nonatomic bitops that are _identical_ to the FD_SET code implemented in asm-*/posix-types.h. Likewise it appears there are other standard-specified bitmask manipulation functions that KABI wants to implement using those. Unfortunately those bitmask functions are only available in kernel space, although a few architectures were having extensive fun with header inclusion loops or implicit inclusion just to get access to them. Unfortunately this made it impossible to use those headers in userspace.
It would be really nice if some of that architecture-specific duplication could be consolidated into a single header file useable from the userspace-pertinent KABI headers _and_ directly from kernel code.
Quoted 1 line
there is another dynamic; those other compilers in general try to emulate gcc to a large degree, usually a very large degree. So the "problem" you see is a lot smaller than you think.
Please read in detail the responses to the non-gcc compiler thread that I mentioned before. Apparently a number of the compilers only take care to conform to the C89 or C99 specifications, which state that such symbols are perfectly ok for them to use, and as a result such "Just like GCC" behavior cannot be relied upon for kernel/userspace-ABI headers.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 3 lines · Kyle Moffett
deal with compatibility: the kernel/userspace boundary. I'd like to strictly define all of that compatibility within a single set of _upstream_ headers in linux/include/linux-kabi or similar.
yes that was what I was hoping you meant
Quoted 3 lines
But returning to your statements above, user code in that (C standards) context means anything not standard-specified.
I think you misread the C standard then. This only applies when you ONLY include the headers defined in the standard, and no others.
Quoted 5 lines
Basically,
from userspace, a program that includes _any_ standard-specified header must bring in *only* those symbols specified by the standard as being present in that header, as well as any implementation- defined double-underscore symbols.
if you follow this argument then ANY library, be it glibc or gtk or mysql or .. can only add __ things. That is not the case. Think of "non-indirect" kernel header includes as including a header for a library.
Maybe an example is needed: include/mtd/mtd-abi.h
Well it's a slightly bad example since it uses stdint.h types not __u64 types. But assume it used __u64 types, then this header would be perfectly fine, including from a standards point of view.
Quoted 3 lines
So unless I "#include
<stdint.h>", my user program can typedefuint32_tto anything it wants, even a struct, if it likes.
yes this the example where __u64 needs underscores, and headers need to use __u64 as they have to today.
Quoted 6 lines
I've actually done that before on
a very strange Cray platform with non-GCC compiler and OS that only supported 64-bit integers (a char was a 64-bit integer with ops to mask out the upper 56 bits), just to make bzip2 work as expected. The same rules apply for _every_ other standard-specified header (stdlib.h,stdio.h,sys/types.h, etc).
exactly. Nowhere is linux/* or asm/* in the standards as specified.
Quoted 2 lines
to get access to them. Unfortunately this made it impossible to use those headers in userspace.
ok now I'm confused. How is providing FUNCTIONALITY such as setting bitops the right thing for such headers? I would argue that the clean headers should absolutely NOT do that but only provide the INTERFACE.
(this is also there for a license angle btw, if you actually start to provide large bodies of code this way the lawyers will get unhappy wrt the GPL; it's a LOT better to strictly do interface things only, eg types, defines and prototypes)
In general:
I think you're focusing WAY too much on the __u64 kind of thing, which is the vast minority of the ABI work, the ioctl structs are the big one. For the __u64 kind of thing I would strongly suggest that you stick to the currently already used symbols; anything existing knows to stay clear of those already, so things are fine there so lets keep it simple for those.
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
In reply to Kyle MoffettHi!
Quoted 21 lines
Quoted 4 lines
Quoted 2 lines
__KABI_everywhere will just make your headers totally unreadable. Please don't do that.Ack, I agree.
Let me reiterate two facts:
(1) The various C standards state that the implementation should restrict itself to symbols prefixed with "__", everything else is reserved for user code (Including symbols prefixed with a single underscore).
(2) GCC predefines a large collection of symbols, macros, and functions for its own use, and this set is not constant (just look at the number of new __-prefixed symbols added between GCC 3 and 4. In addition, we're not just compiling this code under GCC, but people will also be using it (hopefully unmodified) under tiny-cc, intel-cc, PGI, PathScale, Lahey, ARM Ltd, lcc, and possibly others. It probably does not need to be stated that for something as userspace- sensitive as the KABI headers we should not risk colliding with predefined builtins in any of those compilers.
Quoted 6 lines
So my question to the list is this:
Can you come up with any way other than using a "__kabi_" prefix to reasonably avoid namespace collisions with that large list of compilers? If you have some way, I'd be interested to hear it, but as a number of those compilers are commercial I'd have no way to test on them (and I suspect most people on this list would not either).
No, you should just not care about anything but
gcc. intel-cc-version-0.3.2.1.2.5 could use __kabi_struct_dirent or whatever, and collide anyway. By adding __kabi you just make it less likely.
I believe __ is enough. If there's one conflict with some obscure compiler, we can simply fix the conflict (or even fix the compiler :-).
If you feel __ is too dangerous, you may go __k ... It will not look as ugly as __kabi_ , and should be very safe.
Pavel
Signature · 1 line
Picture of sleeping (Linux) penguin wanted...
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 4 lines · Pavel Machek
Quoted 2 lines
So my question to the list is this:
Can you come up with any way other than using a "__kabi_" prefix to reasonably avoid namespace collisions with that large list of compilers? If you have some way, I'd be interested to hear it, but as a number of those compilers are commercial I'd have no way to test on them (and I suspect most people on this list would not either).No, you should just not care about anything but gcc. intel-cc-version-0.3.2.1.2.5 could use
__kabi_struct_direntor whatever, and collide anyway. By adding__kabiyou just make it less likely.
At worst it would just go from "struct dirent" to "struct __kabi_dirent". One reason for this distinction as I believe was highlighted in another email was so that eventually if necessary libc could export a "struct dirent" not the same as the kernel one, and translate between them internally. That would be difficult or impossible now, given the way the kernel exports "struct dirent" directly. I don't remember the specific case where this would have been convenient, but I seem to recall it was mentioned in one of the earlier iterations of this thread.
Quoted 3 lines
I believe __ is enough. If there's one conflict with some obscure compiler, we can simply fix the conflict (or even fix the compiler :-).
If you feel __ is too dangerous, you may go
__k... It will not look as ugly as__kabi_, and should be very safe.
I still disagree with you on this point, but I'll save the arguments for when I have some submittable patches I'd like to get feedback on. I'm also fairly positive that in comparison to the ugliness in some of the necessary C89-compatibility macros, the __kabi_ prefix would be insignificant, but let's leave that discussion for another time as well.
Cheers,
Kyle Moffett
In any case, for reference, here are a few of the specific arguments for support for other compilers:
Quoted 1 line · Daniel Jacobowitz
If you want glibc to ever include these things, they had better be portable C and work without GCC. Otherwise it's a non-starter. Only GCC may be used to build glibc, but it deliberately supports any conforming C compiler to build userspace code.
Quoted 9 lines · Jesper Juhl
Other compilers do exist.
Over the years I've personally used a few to compile userspace apps for different projects (though never for compiling the kernel).
Some of the compilers I have personally used for userspace apps on Linux include: gcc, icc, lcc, tcc Others that I know of but have never used include: sdcc, Compaq C for Linux, Open Watcom, vacpp, XL C/C++
and I'm sure many more exist...
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
In reply to Kyle MoffettQuoted 8 lines · Kyle Moffett
Create initial kernel ABI header infrastructure
Makefile +0
-0
Makefile ifneq ($(KBUILD_SRC),) · line 792
Quoted 6 lines
Makefile +2
-0@@ -787,13 +787,15 @@ ifneq ($(KBUILD_SRC),) /bin/false; \ fi; $(Q)if [ ! -d include2 ]; then mkdir -p include2; fi; + $(Q)if [ ! -d include2/kabi ]; then mkdir -p include2/kabi; fi; $(Q)ln -fsn $(srctree)/include/asm-$(ARCH) include2/asm + $(Q)ln -fsn $(srctree)/include/kabi/arch-$(ARCH) include2/kabi/arch
No - we do not want another symlink.
Create something like:include/i386/kabi-asm/ <= i386 specific filesinclude/kabi/ <= general files
Then we can do:
LINUXINCLUDE += -Iinclude/kabi-$(ARCH)
And the following would work like expected:
#include <kabi/foo.h>
#include <kabi-asm/foo.h>
But this leaves all existing users in the dark cold.
So a more involved approach could be to tkae the opposite approach. To dedicate an area for kernel only header files and make sure this directory is searched _before_ include/
We could do something like
kinclude/linux/ <= generic kernel include headers
kinclude/$(arch)/asm/ <= arch specific include headers
Then adding to the top-level Makefile:
LINUXINCLUDE := -Ikinclude -Ikinclude/$(ARCH)
LINUXINCLUDE += ...
would actually cut it.
Then we would not hurt existing users since they continue to use include/linux/* + include/asm/*
And we could migrate one by one to the kernel clean part.
In other words - a solution that keeps backwards compatibility. A solution that distingush hard between what is the ABI and what is kernel stuff.
And a namespace that is not in use today.
And we have so many users of include/linux today. They do not need a _kabi_ prefix so let it go.
Sam
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 20 lines · Sam Ravnborg
Makefile ifneq ($(KBUILD_SRC),) · line 792
Quoted 13 lines · Kyle Moffett
Create initial kernel ABI header infrastructure
Makefile +2
-0@@ -787,13 +787,15 @@ ifneq ($(KBUILD_SRC),) /bin/false; \ fi; $(Q)if [ ! -d include2 ]; then mkdir -p include2; fi; + $(Q)if [ ! -d include2/kabi ]; then mkdir -p include2/kabi; fi; $(Q)ln -fsn $(srctree)/include/asm-$(ARCH) include2/asm + $(Q)ln -fsn $(srctree)/include/kabi/arch-$(ARCH) include2/kabi/archNo - we do not want another symlink.
Create something like:include/i386/kabi-asm/<= i386 specific filesinclude/kabi/<= general files
Quoted 5 lines
Then we can do:
LINUXINCLUDE += -Iinclude/kabi-$(ARCH)
And the following would work like expected:
#include <kabi/foo.h>
#include <kabi-asm/foo.h>
Hmm, I see where you're going with this. Thanks for the tip!
Quoted 1 line
But this leaves all existing users in the dark cold. So a more involved approach could be to take the opposite approach. To dedicate an area for kernel only header files and make sure this directory is searched _before_
include/
I'm not entirely sure I understand this bit. The idea behind this kabi stuff is precisely to split out portions of the headers so that both userspace and kernelspace can get at them; to designate specific items as "userspace clean" by putting them in kabi; everything else need not care at all, and all those headers would remain in include/linux where they are now. No sense moving _everything_ in include/ around, we just want the parts that userspace needs too.
Quoted 1 line
And we have so many users of
include/linuxtoday. They do not need a _kabi_ prefix so let it go.
_Exactly_ the point. I'd like to leave 99% of the kernel code alone. Userspace doesn't work with the headers in the stock kernel as-is, so I see no problem with working to provide a clean source ABI even if it's partially incompatible with the old source ABI. The binary ABI would stay the same, though, and I'd try to fix the old source ABI where possible.
I think I'll try something like your original proposal:include/kabi/ <= Userspace-clean generic ABI includes
include/$(ARCH)/ <= New arch-specific include directory
include/$(ARCH)/kabi-arch/ <= Arch-specific ABI includes
Then the following Makefile snippet:
LINUXINCLUDE += -Iinclude/$(ARCH)
And then in the header file:
#include <kabi-arch/stddef.h>
#include <kabi/types.h>
It would also be possible to move files from include/asm-$(ARCH) to include/$(ARCH)/asm if desired, but that's a little offtopic for the kernel-ABI headers project I'm working on right now.
Thanks for the advice!
Kyle Moffett
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 34 lines · Kyle Moffett
Quoted 20 lines · Sam Ravnborg
Makefile ifneq ($(KBUILD_SRC),) · line 792
Quoted 13 lines · Kyle Moffett
Create initial kernel ABI header infrastructure
Makefile +2
-0@@ -787,13 +787,15 @@ ifneq ($(KBUILD_SRC),) /bin/false; \ fi; $(Q)if [ ! -d include2 ]; then mkdir -p include2; fi; + $(Q)if [ ! -d include2/kabi ]; then mkdir -p include2/kabi; fi; $(Q)ln -fsn $(srctree)/include/asm-$(ARCH) include2/asm + $(Q)ln -fsn $(srctree)/include/kabi/arch-$(ARCH) include2/kabi/archNo - we do not want another symlink.
Create something like:include/i386/kabi-asm/<= i386 specific filesinclude/kabi/<= general filesQuoted 5 lines
Then we can do:
LINUXINCLUDE += -Iinclude/kabi-$(ARCH)
And the following would work like expected:
#include <kabi/foo.h>
#include <kabi-asm/foo.h>Hmm, I see where you're going with this. Thanks for the tip!
Quoted 4 lines
But this leaves all existing users in the dark cold. So a more involved approach could be to take the opposite approach. To dedicate an area for kernel only header files and make sure this directory is searched _before_
include/
Quoted 8 lines
I'm not entirely sure I understand this bit. The idea behind this kabi stuff is precisely to split out portions of the headers so that both userspace and kernelspace can get at them; to designate specific items as "userspace clean" by putting them in kabi; everything else need not care at all, and all those headers would remain in
include/linux where they are now. No sense moving _everything_ ininclude/around, we just want the parts that userspace needs too.
There are today a great number of users of the existing kernel headers. Introducing a new namespace for the userspace suers will leave them in a dilemma where they have to support kernels before kabi, and kernels with kabi. That alone will limit the acceptance of this.
So shifting to another approach where you take everything that we have today and clean it up step by step will allow us in small steps to go from current mess to a clean ernel ABI.
But we will not have this clean and virgin set of headers until a few years ahead of us.
Keeping include/linux for the kernel ABI will allow us NOT to break existing users and it will allow a stepwise apporach at the same time. But what we make clean and virgin is the kernel specific stuff, not the userspace specific stuff. In final result will be the same but shifting focus to the kernel internal stuff allows us to do ti in incremental steps that does NOT break userspace every and so often.
And we will only break external modules that does not use kbuild to compile - and for hese I really do not care a whit.
Keeping the breakage minimal is the key point here.
Sam
Re: [RFC][PATCH 1/2] Create initial kernel ABI header infrastructure
Quoted 4 lines · Sam Ravnborg
Quoted 1 line · Kyle Moffett
I'm not entirely sure I understand this bit. The idea behind this kabi stuff is precisely to split out portions of the headers so that both userspace and kernelspace can get at them; to designate specific items as "userspace clean" by putting them in kabi; everything else need not care at all, and all those headers would remain in
include/linux where they are now. No sense moving _everything_ ininclude/around, we just want the parts that userspace needs too.There are today a great number of users of the existing kernel headers. Introducing a new namespace for the userspace suers will leave them in a dilemma where they have to support kernels before kabi, and kernels with kabi. That alone will limit the acceptance of this.
I tried to talk to this in my other email, but for emphasis; I'd like to do two things:
1) Preserve compatibility with <linux/*.h>, so that old programs still work fine
2) Cause <kabi/*.h> to work on any distro, and kernel, not just kernel released after it is introduced. In practice this means that a distro could just package up the kabi headers and install them in /usr/include. A package that is altered to compile against only __kabi_* would be able to use information about which syscalls and structures are available in each version to either handle different kernels at runtime or have a configure-time kernel version specified by the user.
Quoted 1 line
Keeping
include/linuxfor the kernel ABI will allow us NOT to break existing users and it will allow a stepwise apporach at the same time.
I completely agree. If you'll notice the patches I submitted were quite careful to preserve the exact same semantics for the linux/*.h headers, both in the interests of not breaking the kernel _and_ in the interests of allowing userspace to still include <linux/types.h>. IMHO, we should try to get the interface compatibility crap _out_ of linux source tree and into a separate package which depends only on the cleaned-up interface specification (IE: kabi/*).
Cheers,
Kyle Moffett
RFC PATCH 2/2 Generalize fd_set handling across architectures
In reply to Kyle MoffettGeneralize fd_set handling across architectures
The fd_set handling is identical across all architectures, but each has its own unique set of ugly functions or macros to do it. This cleans up all of the ugliness and merges the duplicated code into a single KABI header, include/kabi/fdset.h, which is then included in the old places to preserve existing semantics.
Appears to compile correctly with allmodconfig on powerpc (32-bit)
---
commit c00dc3cc75d218b5c23cefcb5324197f34d93de3
tree b4baa9126e0f7490c782279eb069ff184e59a73f
parent affadc51c2b4c96785809a352973a9f6a51a4c37
author Kyle Moffett <[email protected]> Sun, 26 Mar 2006 06:25:07 -0500 committer Kyle Moffett <[email protected]> Sun, 26 Mar 2006 06:25:07 -0500
include/asm-alpha/posix_types.h | 81 +------------------------------------ include/asm-arm/posix_types.h | 22 +--------- include/asm-arm26/posix_types.h | 22 +--------- include/asm-cris/posix_types.h | 23 ----------- include/asm-frv/posix_types.h | 18 +------- include/asm-h8300/posix_types.h | 18 +------- include/asm-i386/posix_types.h | 35 +--------------- include/asm-ia64/posix_types.h | 79 +----------------------------------- include/asm-m32r/posix_types.h | 75 +--------------------------------- include/asm-m68k/posix_types.h | 18 +------- include/asm-mips/posix_types.h | 73 --------------------------------- include/asm-parisc/posix_types.h | 74 +--------------------------------- include/asm-powerpc/posix_types.h | 74 +--------------------------------- include/asm-s390/posix_types.h | 23 +---------- include/asm-sh/posix_types.h | 75 +--------------------------------- include/asm-sh64/posix_types.h | 75 +--------------------------------- include/asm-sparc/posix_types.h | 75 +--------------------------------- include/asm-sparc64/posix_types.h | 75 +--------------------------------- include/asm-v850/posix_types.h | 23 +---------- include/asm-x86_64/posix_types.h | 75 +--------------------------------- include/asm-xtensa/posix_types.h | 68 +------------------------------ include/kabi/fdset.h | 49 ++++++++++++++++++++++ include/linux/fdset.h | 15 +++++++ include/linux/posix_types.h | 35 ---------------- include/linux/time.h | 14 +++--- 25 files changed, 112 insertions(+), 1102 deletions(-)
include/asm-alpha/posix_types.h +2 -79
@@ -1,6 +1,8 @@#ifndef _ALPHA_POSIX_TYPES_H#define _ALPHA_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -41,83 +43,4 @@ typedef __kernel_gid_t __kernel_gid32_t;typedef unsigned int __kernel_old_dev_t ; -#ifdef __KERNEL__ - -#ifndef __GNUC__ - -#define __FD_SET (d, set) ((set)->fds_bits[__FDELT(d)] |= __FDMASK(d)) -#define __FD_CLR (d, set) ((set)->fds_bits[__FDELT(d)] &= ~__FDMASK(d)) -#define __FD_ISSET (d, set) (((set)->fds_bits[__FDELT(d)] & __FDMASK(d)) != 0) -#define __FD_ZERO (set) \ - ((void) memset ((__ptr_t) (set), 0, sizeof (__kernel_fd_set))) - -#else /* __GNUC__ */ - -/* With GNU C, use inline functions instead so args are evaluated only once: */ - -#undef __FD_SET -static __inline__void __FD_SET (unsigned long fd,__kernel_fd_set *fdsetp) -{ -unsigned long _tmp = fd / __NFDBITS; -unsigned long _rem = fd % __NFDBITS; - fdsetp->fds_bits[_tmp] |= (1UL <<_rem); -} - -#undef __FD_CLR -static __inline__void __FD_CLR (unsigned long fd,__kernel_fd_set *fdsetp) -{ -unsigned long _tmp = fd / __NFDBITS; -unsigned long _rem = fd % __NFDBITS; - fdsetp->fds_bits[_tmp] &= ~(1UL <<_rem); -} - -#undef __FD_ISSET -static __inline__int __FD_ISSET (unsigned long fd,const __kernel_fd_set *p) -{ -unsigned long _tmp = fd / __NFDBITS; -unsigned long _rem = fd % __NFDBITS; -return (p->fds_bits[_tmp] & (1UL <<_rem)) !=0 ; -} - -/* -* This will unroll the loop for the normal constant case (8 ints, -* for a 256-bit fd_set) -*/ -#undef __FD_ZERO -static __inline__void __FD_ZERO (__kernel_fd_set *p) -{ -unsigned long *tmp = p->fds_bits; -int i; - -if (__builtin_constant_p (__FDSET_LONGS)) { -switch (__FDSET_LONGS) { -case 16 : - tmp[0 ] =0 ; tmp[1 ] =0 ; tmp[2 ] =0 ; tmp[3 ] =0 ; - tmp[4 ] =0 ; tmp[5 ] =0 ; tmp[6 ] =0 ; tmp[7 ] =0 ; - tmp[8 ] =0 ; tmp[9 ] =0 ; tmp[10 ] =0 ; tmp[11 ] =0 ; - tmp[12 ] =0 ; tmp[13 ] =0 ; tmp[14 ] =0 ; tmp[15 ] =0 ; -return ; - -case 8 : - tmp[0 ] =0 ; tmp[1 ] =0 ; tmp[2 ] =0 ; tmp[3 ] =0 ; - tmp[4 ] =0 ; tmp[5 ] =0 ; tmp[6 ] =0 ; tmp[7 ] =0 ; -return ; - -case 4 : - tmp[0 ] =0 ; tmp[1 ] =0 ; tmp[2 ] =0 ; tmp[3 ] =0 ; -return ; - } - } - i = __FDSET_LONGS; -while (i) { - i--; - *tmp =0 ; - tmp++; - } -} - -#endif /* __GNUC__ */ - -#endif /* __KERNEL__ */ -#endif /* _ALPHA_POSIX_TYPES_H */
include/asm-arm/posix_types.h +2 -20
@@ -13,6 +13,8 @@#ifndef __ARCH_ARM_POSIX_TYPES_H#define __ARCH_ARM_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -58,24 +60,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (fd, fdsetp) \ - (((fd_set *)(fdsetp))->fds_bits[(fd) >> 5] |= (1<<((fd) & 31))) - -#undef __FD_CLR -#define __FD_CLR (fd, fdsetp) \ - (((fd_set *)(fdsetp))->fds_bits[(fd) >> 5] &= ~(1<<((fd) & 31))) - -#undef __FD_ISSET -#define __FD_ISSET (fd, fdsetp) \ - ((((fd_set *)(fdsetp))->fds_bits[(fd) >> 5] & (1<<((fd) & 31))) != 0) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) \ - (memset (fdsetp, 0, sizeof (*(fd_set *)(fdsetp)))) - -#endif -#endif
include/asm-arm26/posix_types.h +2 -20
@@ -13,6 +13,8 @@#ifndef __ARCH_ARM_POSIX_TYPES_H#define __ARCH_ARM_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -58,24 +60,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (fd, fdsetp) \ - (((fd_set *)(fdsetp))->fds_bits[(fd) >> 5] |= (1<<((fd) & 31))) - -#undef __FD_CLR -#define __FD_CLR (fd, fdsetp) \ - (((fd_set *)(fdsetp))->fds_bits[(fd) >> 5] &= ~(1<<((fd) & 31))) - -#undef __FD_ISSET -#define __FD_ISSET (fd, fdsetp) \ - ((((fd_set *)(fdsetp))->fds_bits[(fd) >> 5] & (1<<((fd) & 31))) != 0) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) \ - (memset ((fdsetp), 0, sizeof (*(fd_set *)(fdsetp)))) - -#endif -#endif
include/asm-cris/posix_types.h +1 -22
@@ -1,12 +1,9 @@/* $Id: posix_types.h,v 1.1 2000/07/10 16:32:31 bjornw Exp $ */ -/* We cheat a bit and use our C-coded bitops functions from asm/bitops.h */ -/* I guess we should write these in assembler because they are used often. */ -#ifndef __ARCH_CRIS_POSIX_TYPES_H#define __ARCH_CRIS_POSIX_TYPES_H -#include <asm/bitops.h> +#include <linux/fdset.h> /* * This file is generally used by user-level software, so you need to
@@ -53,22 +50,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -/* should this ifdef be here ? */ - -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (fd,fdsetp) set_bit(fd, (void *)(fdsetp)) - -#undef __FD_CLR -#define __FD_CLR (fd,fdsetp) clear_bit(fd, (void *)(fdsetp)) - -#undef __FD_ISSET -#define __FD_ISSET (fd,fdsetp) test_bit(fd, (void *)(fdsetp)) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) memset((void *)(fdsetp), 0, __FDSET_LONGS << 2) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2) */ -#endif /* __ARCH_CRIS_POSIX_TYPES_H */
include/asm-frv/posix_types.h +2 -16
@@ -1,6 +1,8 @@#ifndef _ASM_POSIX_TYPES_H#define _ASM_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -46,21 +48,5 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (d, set) ((set)->fds_bits[__FDELT(d)] |= __FDMASK(d)) - -#undef __FD_CLR -#define __FD_CLR (d, set) ((set)->fds_bits[__FDELT(d)] &= ~__FDMASK(d)) - -#undef __FD_ISSET -#define __FD_ISSET (d, set) (!!((set)->fds_bits[__FDELT(d)] & __FDMASK(d))) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) (memset (fdsetp, 0, sizeof(*(fd_set *)fdsetp))) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2) */ -#endif
include/asm-h8300/posix_types.h +2 -16
@@ -1,6 +1,8 @@#ifndef __ARCH_H8300_POSIX_TYPES_H#define __ARCH_H8300_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -45,20 +47,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (d, set) ((set)->fds_bits[__FDELT(d)] |= __FDMASK(d)) - -#undef __FD_CLR -#define __FD_CLR (d, set) ((set)->fds_bits[__FDELT(d)] &= ~__FDMASK(d)) - -#undef __FD_ISSET -#define __FD_ISSET (d, set) ((set)->fds_bits[__FDELT(d)] & __FDMASK(d)) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) (memset (fdsetp, 0, sizeof(*(fd_set *)fdsetp))) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2) */ -#endif
include/asm-i386/posix_types.h +2 -33
@@ -1,6 +1,8 @@#ifndef __ARCH_I386_POSIX_TYPES_H#define __ARCH_I386_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -46,37 +48,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (fd,fdsetp) \ - __asm__ __volatile__("btsl %1,%0": \ - "=m" (*(__kernel_fd_set *) (fdsetp)):"r" ((int) (fd))) - -#undef __FD_CLR -#define __FD_CLR (fd,fdsetp) \ - __asm__ __volatile__("btrl %1,%0": \ - "=m" (*(__kernel_fd_set *) (fdsetp)):"r" ((int) (fd))) - -#undef __FD_ISSET -#define __FD_ISSET (fd,fdsetp) (__extension__ ({ \ - unsigned char __result; \ - __asm__ __volatile__("btl %1,%2 ; setb %0" \ - :"=q" (__result) :"r" ((int) (fd)), \ - "m" (*(__kernel_fd_set *) (fdsetp))); \ - __result; })) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) \ -do { \ - int __d0, __d1; \ - __asm__ __volatile__("cld ; rep ; stosl" \ - :"=m" (*(__kernel_fd_set *) (fdsetp)), \ - "=&c" (__d0), "=&D" (__d1) \ - :"a" (0), "1" (__FDSET_LONGS), \ - "2" ((__kernel_fd_set *) (fdsetp)) : "memory"); \ -} while (0) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2) */ -#endif
include/asm-ia64/posix_types.h +2 -77
89 lines not on this page · raw message
include/asm-m32r/posix_types.h +2 -73
85 lines not on this page · raw message
include/asm-m68k/posix_types.h +2 -16
@@ -1,6 +1,8 @@#ifndef __ARCH_M68K_POSIX_TYPES_H#define __ARCH_M68K_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -46,20 +48,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL) */ } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#undef __FD_SET -#define __FD_SET (d, set) ((set)->fds_bits[__FDELT(d)] |= __FDMASK(d)) - -#undef __FD_CLR -#define __FD_CLR (d, set) ((set)->fds_bits[__FDELT(d)] &= ~__FDMASK(d)) - -#undef __FD_ISSET -#define __FD_ISSET (d, set) ((set)->fds_bits[__FDELT(d)] & __FDMASK(d)) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) (memset (fdsetp, 0, sizeof(*(fd_set *)fdsetp))) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2) */ -#endif
include/asm-mips/posix_types.h +1 -72
83 lines not on this page · raw message
include/asm-parisc/posix_types.h +2 -72
84 lines not on this page · raw message
include/asm-powerpc/posix_types.h +2 -72
84 lines not on this page · raw message
include/asm-s390/posix_types.h +2 -21
@@ -9,6 +9,8 @@#ifndef __ARCH_S390_POSIX_TYPES_H#define __ARCH_S390_POSIX_TYPES_H +#include <linux/fdset.h> +/* * This file is generally used by user-level software, so you need to * be a little careful about namespace pollution etc. Also, we cannot
@@ -75,25 +77,4 @@ typedef struct { #endif /* !defined(__KERNEL__) && !defined(__USE_ALL)*/ } __kernel_fsid_t; - -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -#ifndef _S390_BITOPS_H -#include <asm/bitops.h> -#endif - -#undef __FD_SET -#define __FD_SET (fd,fdsetp) set_bit((fd),(fdsetp)->fds_bits) - -#undef __FD_CLR -#define __FD_CLR (fd,fdsetp) clear_bit((fd),(fdsetp)->fds_bits) - -#undef __FD_ISSET -#define __FD_ISSET (fd,fdsetp) test_bit((fd),(fdsetp)->fds_bits) - -#undef __FD_ZERO -#define __FD_ZERO (fdsetp) (memset ((fdsetp), 0, sizeof(*(fd_set *)(fdsetp)))) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2)*/ -#endif
include/asm-sh/posix_types.h +2 -73
85 lines not on this page · raw message
include/asm-sh64/posix_types.h +2 -73
85 lines not on this page · raw message
include/asm-sparc/posix_types.h +2 -73
85 lines not on this page · raw message
include/asm-sparc64/posix_types.h +2 -73
85 lines not on this page · raw message
include/asm-v850/posix_types.h +2 -21
@@ -14,6 +14,8 @@#ifndef __V850_POSIX_TYPES_H__#define __V850_POSIX_TYPES_H__ +#include <linux/fdset.h> +typedef unsigned long __kernel_ino_t ;typedef unsigned long long __kernel_ino64_t ;typedef unsigned int __kernel_mode_t ;
@@ -52,25 +54,4 @@ typedef struct { } __kernel_fsid_t; -#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ <2 ) - -/* We used to include <asm/bitops.h> here, which seems the right thing, but -it caused nasty include-file definition order problems. Removing the -include seems to work, so fingers crossed... */ - -#undef __FD_SET -#define __FD_SET (fd, fd_set) \ - __set_bit (fd, (void *)&((__kernel_fd_set *)fd_set)->fds_bits) -#undef __FD_CLR -#define __FD_CLR (fd, fd_set) \ - __clear_bit (fd, (void *)&((__kernel_fd_set *)fd_set)->fds_bits) -#undef __FD_ISSET -#define __FD_ISSET (fd, fd_set) \ - __test_bit (fd, (void *)&((__kernel_fd_set *)fd_set)->fds_bits) -#undef __FD_ZERO -#define __FD_ZERO (fd_set) \ - memset (fd_set, 0, sizeof (*(fd_set *)fd_set)) - -#endif /* defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2) */ -#endif /* __V850_POSIX_TYPES_H__ */
include/asm-x86_64/posix_types.h +2 -73
85 lines not on this page · raw message
include/asm-xtensa/posix_types.h +2 -66
78 lines not on this page · raw message
include/kabi/fdset.h new file +49 -0
49 lines not on this page · raw message
include/linux/fdset.h new file +15 -0
@@ -0,0 +1,15 @@ +#ifndef __LINUX_FDSET_H +# define __LINUX_FDSET_H 1 + +# include <kabi/fdset.h> + +# define __NFDBITS __KABI_NFDBITS +# define __FD_SETSIZE __KABI_FD_SETSIZE +# define __FD_SET (fd,set) __KABI_FD_SET(fd,set) +# define __FD_CLR (fd,set) __KABI_FD_CLR(fd,set) +# define __FD_ISSET (fd,set) __KABI_FD_ISSET(fd,set) +# define __FD_ZERO (set) __KABI_FD_ZERO(set) + +typedef __kabi_fd_set __kernel_fd_set ; + +#endif /* not __LINUX_FDSET_H */
include/linux/posix_types.h +1 -34
@@ -2,40 +2,7 @@#define _LINUX_POSIX_TYPES_H#include <linux/stddef.h> - -/* -* This allows for 1024 file descriptors: if NR_OPEN is ever grown -* beyond that you'll have to change this too. But 1024 fd's seem to be -* enough even for such "real" unices like OSF/1, so hopefully this is -* one limit that doesn't have to be changed [again]. -* -* Note that POSIX wants the FD_CLEAR(fd,fdsetp) defines to be in -* <sys/time.h> (and thus <linux/time.h>) - but this is a more logical -* place for them. Solved by having dummy defines in <sys/time.h>. -*/ - -/* -* Those macros may have been defined in <gnu/types.h>. But we always -* use the ones here. -*/ -#undef __NFDBITS -#define __NFDBITS (8 * sizeof(unsigned long)) - -#undef __FD_SETSIZE -#define __FD_SETSIZE 1024 - -#undef __FDSET_LONGS -#define __FDSET_LONGS (__FD_SETSIZE/__NFDBITS) - -#undef __FDELT -#define __FDELT (d) ((d) / __NFDBITS) - -#undef __FDMASK -#define __FDMASK (d) (1UL << ((d) % __NFDBITS)) - -typedef struct { -unsigned long fds_bits [__FDSET_LONGS]; -}__kernel_fd_set ; +#include <linux/fdset.h> /* Type of a signal handler. */ typedef void (*__kernel_sighandler_t )(int );
include/linux/time.h +7 -7
@@ -150,13 +150,13 @@ extern struct timeval ns_to_timeval(cons#endif /* __KERNEL__ */ -#define NFDBITS __NFDBITS - -#define FD_SETSIZE __FD_SETSIZE -#define FD_SET (fd,fdsetp) __FD_SET(fd,fdsetp) -#define FD_CLR (fd,fdsetp) __FD_CLR(fd,fdsetp) -#define FD_ISSET (fd,fdsetp) __FD_ISSET(fd,fdsetp) -#define FD_ZERO (fdsetp) __FD_ZERO(fdsetp) +#include <kabi/fdset.h> +#define NFDBITS __KABI_NFDBITS +#define FD_SETSIZE __KABI_FD_SETSIZE +#define FD_SET (fd,set) __KABI_FD_SET(fd,set) +#define FD_CLR (fd,set) __KABI_FD_CLR(fd,set) +#define FD_ISSET (fd,set) __KABI_FD_ISSET(fd,set) +#define FD_ZERO (set) __KABI_FD_ZERO(set)/* * Names of the interval timers, and structure
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 3 lines · On Sun, 26 Mar 2006 06:52:05 -0500 Kyle Moffett <[email protected]> wrote:
2) Since most of the headers are currently quite broken with respect to GLIBC and userspace, I won't spend much extra time preserving compatibility with GLIBC, userspace, or non-GCC compilers.
That didn't come out right, but what I meant to say was this: Since the headers in include/linux are quite broken with respect to GLIBC and userspace, I won't let so-called "compatibility" code like this get in the way:
#ifndef __GNUC__
#define DO_SOMETHING(foo) ICKY_MACRO
#else
static __inline__ void DO_SOMETHING(int foo)
{
sensible_inline_function();
}
#endif
You can see where I take that approach in the patches I sent.
One other thing I would like to point out: The fd_set code wants to use __set_bit and __clear_bit from <linux/bitops.h>, but those really should not be accessible to userspace directly. I would like to propose moving that functionality into <__klib/*.h> from which it would be accessible to both <linux/*.h> and <kabi/*.h>. I think this would also help with the UML header issues by providing those kernel-internal APIs to the kernel when run from userspace. (Please correct me if I'm wrong).
I appreciate your comments and corrections, thanks!
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 9 lines · Kyle Moffett
Quoted 3 lines · On Sun, 26 Mar 2006 06:52:05 -0500 Kyle Moffett <[email protected]> wrote:
2) Since most of the headers are currently quite broken with respect to GLIBC and userspace, I won't spend much extra time preserving compatibility with GLIBC, userspace, or non-GCC compilers.
That didn't come out right, but what I meant to say was this: Since the headers in
include/linuxare quite broken with respect to GLIBC and userspace, I won't let so-called "compatibility" code like this get in the way:
As long as the eventual goal is something that *works* with glibc and uClibc and the rest of them, that's good enough as far as I can tell. Obviously compatibility crud *in the headers* which serves no obvious purpose and which userspace doesn't actually need should die (and there's certainly some like that lying around).
Signature · 3 lines
`Come now, you should know that whenever you plan the duration of your unplanned downtime, you should add in padding for random management freakouts.'
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 6 lines · Kyle Moffett
Quoted 2 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
is KABI the right name? I mean.. from the kernel pov it's the interface to userspace ;)
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 10 lines · Arjan van de Ven
Quoted 6 lines · Kyle Moffett
Quoted 2 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
is KABI the right name? I mean.. from the kernel pov it's the interface to userspace ;)
ok proposal: just use "abi" without the K; it is THE abi, and the only one the kernel has, so no need for prefixes or suffixes.
(BTW I'm 100% behind the idea of a clean set of seperate ABI headers)
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Arjan van de VenQuoted 7 lines · Arjan van de Ven
Quoted 4 lines · Kyle Moffett
Quoted 1 line · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
is KABI the right name? I mean.. from the kernel pov it's the interface to userspace ;)
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 11 lines · Kyle Moffett
Quoted 10 lines · Arjan van de Ven
Quoted 8 lines · Kyle Moffett
Quoted 2 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
Quoted 8 lines
Quoted 2 lines
is KABI the right name? I mean.. from the kernel pov it's the interface to userspace ;)
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
Well, since I guess UAKDABI (User Accessible Kernel Defined) is out of the question, what about LABI? (Linux ABI)
Signature · 5 lines
Giuseppe "Oblomov" Bilotta Axiom I of the Giuseppe Bilotta theory of IT: Anything is better than MS
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 3 lines · Kyle Moffett
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
How about __linux, or __linux_abi? There are ABIs for other components, and other OSes. Linux is the name of the project after all.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 4 lines · Avi Kivity
Quoted 1 line · Kyle Moffett
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
How about
__linux, or__linux_abi? There are ABIs for other components, and other OSes. Linux is the name of the project after all.
The other thing that I quickly noticed while writing up the patches is that it's kind of tedious typing __kabi_ over and over again. I actually did first try with __linux_abi_ but the typing effort and finger cramps made me give up on that really quickly.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 4 lines · Kyle Moffett
The other thing that I quickly noticed while writing up the patches is that it's kind of tedious typing
__kabi_over and over again. I actually did first try with__linux_abi_but the typing effort and finger cramps made me give up on that really quickly.
What kind of barebone editor are you using that doesn't offer a way (through macro or automatic completion or abbreviations or substitution or plain copy and paste or whatever else) to shorten that typing effort?
Signature · 8 lines
Giuseppe "Oblomov" Bilotta [W]hat country can preserve its liberties, if its rulers are not warned from time to time that [the] people preserve the spirit of resistance? Let them take arms...The tree of liberty must be refreshed from time to time, with the blood of patriots and tyrants. -- Thomas Jefferson, letter to Col. William S. Smith, 1787
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 4 lines · Giuseppe Bilotta
Quoted 1 line · Kyle Moffett
The other thing that I quickly noticed while writing up the patches is that it's kind of tedious typing
__kabi_over and over again. I actually did first try with__linux_abi_but the typing effort and finger cramps made me give up on that really quickly.What kind of barebone editor are you using that doesn't offer a way (through macro or automatic completion or abbreviations or substitution or plain copy and paste or whatever else) to shorten that typing effort?
Well, actually I was using vim, and although I know how to configure shortcuts like that; do we really want to force people to use shortcuts and abbreviations to be able to sanely edit the ABI code? Isn't that counterproductive with the desire to get the maintainers fixing up their own ABIs? Plus all the extra verbiage tends to make the lines too long (80-column limit) and makes it harder to read.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 12 lines · Kyle Moffett
Quoted 10 lines · Giuseppe Bilotta
Quoted 4 lines · Kyle Moffett
The other thing that I quickly noticed while writing up the patches is that it's kind of tedious typing
__kabi_over and over again. I actually did first try with__linux_abi_but the typing effort and finger cramps made me give up on that really quickly.What kind of barebone editor are you using that doesn't offer a way (through macro or automatic completion or abbreviations or substitution or plain copy and paste or whatever else) to shorten that typing effort?
Quoted 6 lines
Well, actually I was using vim, and although I know how to configure shortcuts like that; do we really want to force people to use shortcuts and abbreviations to be able to sanely edit the ABI code? Isn't that counterproductive with the desire to get the maintainers fixing up their own ABIs? Plus all the extra verbiage tends to make the lines too long (80-column limit) and makes it harder to read.
Oh, I'm not exactly a fan of verbosity myself (essentially for the lines too long and hard to read), which is why I suggested __labi_ as a prefix --I wasn't arguing in favour of __linux_abi_.
However, as far as I know all programmer-friendly editors have some form of easy to use code-completion feature (e.g. in vim ctrl+n/ctrl+p are very handy without the need to define specific shortcuts, and the new completion code in the future vim 7 is extremely promising ...) and while I do agree that __linux_abi_ is too long, you remarked yourself __[ukl]abi_ is still annoying, albeit shorter :) *Any* prefix chosen will benefit from completion ;)
Signature · 5 lines
Giuseppe "Oblomov" Bilotta "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." Benjamin Franklin
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 7 lines · Kyle Moffett
Quoted 4 lines · Avi Kivity
Quoted 1 line · Kyle Moffett
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
How about
__linux, or__linux_abi? There are ABIs for other components, and other OSes. Linux is the name of the project after all.The other thing that I quickly noticed while writing up the patches is that it's kind of tedious typing
__kabi_over and over again. I actually did first try with__linux_abi_but the typing effort and finger cramps made me give up on that really quickly.
#define _LA(x) __linux_abi_##x
struct _LA(whatever) {
int foo;
int bar;
};
struct _LA(another) {
...
};
#undef _LA
?
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 18 lines · Avi Kivity
Quoted 8 lines · Kyle Moffett
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Kyle Moffett
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
Quoted 7 lines
Quoted 2 lines
How about
__linux, or__linux_abi? There are ABIs for other components, and other OSes. Linux is the name of the project after all.The other thing that I quickly noticed while writing up the patches is that it's kind of tedious typing
__kabi_over and over again. I actually did first try with__linux_abi_but the typing effort and finger cramps made me give up on that really quickly.#define
_LA(x)__linux_abi_##x
Quoted 8 lines
struct _LA(whatever) { int foo; int bar; };struct _LA(another) { ... };
this is a good sign that this is all very over designed :)
namespace pollution is perhaps evil, but we also should not overreact. Especially for struct names. *IF* they are in a "narrow enough" header, the user of the header knows what he is doing, and accepts these to be in his namespace.
The problem is things like u64 etc that is VERY common in all headers, but then again __u64 etc are just fine, history has proven that already.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 5 lines · Arjan van de Ven
Quoted 5 lines
struct _LA(whatever) { int foo; int bar; };
Quoted 8 lines
Quoted 4 lines
struct _LA(another) { ... };this is a good sign that this is all very over designed :)
It's an eyesore, isn't it? :)
Quoted 5 lines
namespace pollution is perhaps evil, but we also should not overreact. Especially for struct names. *IF* they are in a "narrow enough" header, the user of the header knows what he is doing, and accepts these to be in his namespace.
This is true for a small enough application. But things grow, libraries are added, and includes keep pulling other includes in. Sooner or later you'll have a collision.
Quoted 3 lines
The problem is things like u64 etc that is VERY common in all headers, but then again
__u64etc are just fine, history has proven that already.
Agree. But to be on the safe side one can use uint64_t and friends (which the kernel can typedef to u64 and first degree relatives)
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
\
Quoted 8 lines
This is true for a small enough application. But things grow, libraries are added, and includes keep pulling other includes in. Sooner or later you'll have a collision.
Quoted 3 lines
The problem is things like u64 etc that is VERY common in all headers, but then again
__u64etc are just fine, history has proven that already.Agree. But to be on the safe side one can use
uint64_tand friends (which the kernel can typedef to u64 and first degree relatives)
actually uint64_t is a bigger namespace pollution than __u64 ;)
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Avi KivityQuoted 3 lines · Avi Kivity
This is true for a small enough application. But things grow, libraries are added, and includes keep pulling other includes in. Sooner or later you'll have a collision.
And you'll fix it when it happens. Fact of life.
Quoted 5 lines
Quoted 2 lines
The problem is things like u64 etc that is VERY common in all headers, but then again
__u64etc are just fine, history has proven that already.Agree. But to be on the safe side one can use
uint64_tand friends (which the kernel can typedef to u64 and first degree relatives)
Now that the kernel no longer targets gcc before 3.2, c99 names are merely ugly rather than an actual problem. :)
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 5 lines · Rob Landley
Quoted 3 lines
Quoted 1 line
The problem is things like u64 etc that is VERY common in all headers, but then again
__u64etc are just fine, history has proven that already.Agree. But to be on the safe side one can use
uint64_tand friends (which the kernel can typedef to u64 and first degree relatives)Now that the kernel no longer targets gcc before 3.2, c99 names are merely ugly rather than an actual problem. :)
No, they're still a problem in ABI headers because they have defined visibility. A userspace program that does not include <stdint.h> will not have those types put in its namespace. Including <sys/time.h> _cannot_ bring <stdint.> in automatically, that breaks the POSIX symbol visibility rules.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Rob LandleyQuoted 9 lines · Rob Landley
Quoted 4 lines · Avi Kivity
This is true for a small enough application. But things grow, libraries are added, and includes keep pulling other includes in. Sooner or later you'll have a collision.
And you'll fix it when it happens. Fact of life.
Fixing it will mean breaking either the ABI of the kernel or of the large library you pulled in.
An ABI bug causes pain far beyond its size. Look at the trouble caused when some interfaces uses unsigned instead of u64. In kernel APIs, you just replace the type, but in the ABI, you add a new syscall or do some other hack.
Much better to get it right the first time, even if it's ugly. It's an ABI, not a beauty contest nominee.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Rob Landley
Quoted 3 lines · Avi Kivity
This is true for a small enough application. But things grow, libraries are added, and includes keep pulling other includes in. Sooner or later you'll have a collision.
And you'll fix it when it happens. Fact of life.
Quoted 7 lines
Fixing it will mean breaking either the ABI of the kernel or of the large library you pulled in.
An ABI bug causes pain far beyond its size. Look at the trouble caused when some interfaces uses unsigned instead of u64. In kernel APIs, you just replace the type, but in the ABI, you add a new syscall or do some other hack.
One of busybox's users has a libc built with improperly sanitized kernel headers that leak all the kernel's CONFIG_ symbols into the standard namespace, and we found out because one of them clashes with a busybox CONFIG symbol and it broke his build.
The easiest thing to do was rename our CONFIG symbol, which we did. (And asked him to fix his system, which he didn't.) This was a while ago now...
Quoted 2 lines
Much better to get it right the first time, even if it's ugly. It's an ABI, not a beauty contest nominee.
I've been cc'd on all this because I cared about Mazur's kernel headers enough to email him several times. But I can't say I really care about this new project that's taken over the discussion.
Either it doesn't break the existing interface, in which case it has no impact on me and I can ignore it, or it requires all existing userspace programs that care about the current state of things to change to care about your new thing instead, in which case it's probably doomed. Unless the kernel picks it up and imposes a flag day, at which point I'll find out about it from LWN like everybody else.
Either way, it's not sounding like something I can grab and build uClibc systems with any time soon, the way I could use Mazur's headers to build uClibc. I'll probably wind up using the gentoo headers when the 2.6.14 version ships.
http://www.gentoo.org/cgi-bin/viewcvs.cgi/src/patchsets/gentoo-headers/?root=gentoo
*shrug*. Good luck with whatever it is you're trying to accomplish.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 4 lines · Rob Landley
Either way, it's not sounding like something I can grab and build uClibc systems with any time soon, the way I could use Mazur's headers to build uClibc. I'll probably wind up using the gentoo headers when the 2.6.14 version ships.
That's the trouble. While I have nothing against someone (as in -- not me :) doing all that abi separation, it's not something I can use straight away. Hell, I don't even get where in all of it I'd end up with something I could use (granted, I haven't looked into the thread too closely). Not to mention, that I suspect, that if there were enough people to do it, it would have gotten done two years ago.
So unless anybody's got a better idea, I'll try releasing some initial version of that llh-ng thingie rather soonish and see how that'll work out. Anybody with me on that? :)
Signature · 5 lines
Judge others by their intentions and yourself by your results.
Guy Kawasaki
Education is an admirable thing, but it is well to remember from
time to time that nothing that is worth knowing can be taught.
Oscar WildeRe: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 6 lines · Mariusz Mazur
Quoted 1 line · Rob Landley
Either way, it's not sounding like something I can grab and build uClibc systems with any time soon, the way I could use Mazur's headers to build uClibc. I'll probably wind up using the gentoo headers when the 2.6.14 version ships.
That's the trouble. While I have nothing against someone (as in -- not me :) doing all that abi separation, it's not something I can use straight away. Hell, I don't even get where in all of it I'd end up with something I could use (granted, I haven't looked into the thread too closely). Not to mention, that I suspect, that if there were enough people to do it, it would have gotten done two years ago.
So unless anybody's got a better idea, I'll try releasing some initial version of that llh-ng thingie rather soonish and see how that'll work out. Anybody with me on that? :)
I agree with this approach. I don't think the llh-ng method will be maintainable in the long-term, which is why I'm doing the KABI patches, but I think that in the short term that's the fastest way to get a working set of headers against which to build glibc.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Mariusz MazurQuoted 13 lines · Mariusz Mazur
Quoted 4 lines · Rob Landley
Either way, it's not sounding like something I can grab and build uClibc systems with any time soon, the way I could use Mazur's headers to build uClibc. I'll probably wind up using the gentoo headers when the 2.6.14 version ships.
That's the trouble. While I have nothing against someone (as in -- not me :) doing all that abi separation, it's not something I can use straight away. Hell, I don't even get where in all of it I'd end up with something I could use (granted, I haven't looked into the thread too closely). Not to mention, that I suspect, that if there were enough people to do it, it would have gotten done two years ago.
Quoted 3 lines
So unless anybody's got a better idea, I'll try releasing some initial version of that llh-ng thingie rather soonish and see how that'll work out. Anybody with me on that? :)
I'm highly interested in seeing the result. :)
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 5 lines · Kyle Moffett
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
Ok, question, using an example I'm familiar with. What's affected by the prefix?
The busybox code for associating loopback devices with files was a problem for _years_. The standard advice was not to include linux/loop.h, but instead cut and paste the structure out of the header into your own code. (Apparently, you're allowed to be uncomfortable if somebody suggests that this is the right way to get portability between bsd/cygwin/linux, but between 2.4 and 2.6 it's best practices. Ok.)
Unfortunately, the cut and paste approach turned out to be incredibly painful because busybox needs to be portable between hardware platforms and between kernel versions.
Problem 1: The actual type of __kernel_dev_t varies between x86/arm/alpha/ppc/etc. Any attempt to define our own loop ioctl structure meant we needed to either use the kernel-defined data type for this, or have an #ifdef forest for every possible platform somebody might want to build busybox on. (This was exploded into a mess instantly.)
But using the kernel kdev_t type sucked too, because between 2.4 and 2.6 it got renamed to old_kdev_t so then we needed #ifdefs to check the kernel version of the headers we'd imported to see which name it was using. (And it still broke anyway in strange various corner cases depending on which distro people were building under.)
We wandered through dozens of different approaches trying to avoid "#include <linux/loop.h>". We had #ifdefs for glibc and #ifdefs for uclibc... It was terrible. About half the suggestions got checked into the tree at various points:
http://www.busybox.net/cgi-bin/viewcvs.cgi/trunk/busybox/libbb/loop.c?rev=14554&view=log
There were more that merely showed up on the list...
We eventually got it all untangled. Now we check if we're on 2.6 and if so use the 64 bit API (which is stable on 2.6), and otherwise use a copy of the 32 bit structure that predates __kernel_dev_t being renamed. And we trust the darn kernel headers to supply the types for us, because there's just no alternative.
My question is: will your new approach work with our existing code, or are you saying we need another #ifdef to check for yet another gratuitously incompatible name, ala _kabi__kernel_dev_t? Because I _really_ don't want to go there. Nobody will be able to build existing versions of busybox (at least the mount/umount/losetup commands) against your new thing. New versions that actually depended on _kabi_blah wouldn't build on any older distros unless we add another #ifdef...
This is just an example. There is currently no existing program out in userspace that is tries to talk to the kernel using names with _kabi_ prepended to them, that I am aware of. If you're adding this prefix, you're writing a brand new interface which currently has exactly zero users. Is this the intent?
Quoted 2 lines
Cheers,
Kyle Moffett
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 8 lines · Rob Landley
Quoted 1 line · Kyle Moffett
Well I guess you could call it UABI, but that might also imply that it's _userspace_ that defines the interface, instead of the kernel. Since the headers themselves are rather tightly coupled with the kernel, I think I'll stick with the KABI name for now (unless somebody can come up with a better one, of course :-D).
Ok, question, using an example I'm familiar with. What's affected by the prefix?
We eventually got it all untangled. Now we check if we're on 2.6 and if so use the 64 bit API (which is stable on 2.6), and otherwise use a copy of the 32 bit structure that predates
__kernel_dev_tbeing renamed. And we trust the darn kernel headers to supply the types for us, because there's just no alternative.My question is: will your new approach work with our existing code, or are you saying we need another #ifdef to check for yet another gratuitously incompatible name, ala
_kabi__kernel_dev_t?
No, no need for a new #ifdef, at least not for a long time if you don't want to. My plans for the immediate future are only to touch files that are currently broken with respect to !__KERNEL__, and linux/loop.h isn't. When I do get to it, my plan is to bring the parts that userspace needs into kabi/loop.h, named as __kabi_*, however linux/loop.h will define the new symbols to their old names for compatibility with linux code and userspace code that happens to be using those interfaces, so you still won't need to change anything. Hopefully this would also mean that if the shipped linux/loop.h is currently broken for your uses, I would be able to fix it so you don't need a customized copy at all.
If you _did_ decide to switch busybox to use the __kabi_ symbols, you would be able to take advantage of the fact that the same set of kabi headers (which don't include kernelversion.h) would work unmodified for kernels all the way back to 0.99, as long as you base your decisions of what features are available on either a dynamically-detected version or a pre-configured minimum version from the user. I'm assuming here that it's possible to compile a busybox that works on a 2.2 or 2.4 kernel even using linux-libc-headers 2.6.*, as long as the user indicates such.
Quoted 1 line
This is just an example. There is currently no existing program out in userspace that is tries to talk to the kernel using names with _kabi_ prepended to them, that I am aware of. If you're adding this prefix, you're writing a brand new interface which currently has exactly zero users. Is this the intent?
I'd like this project to have a twofold purpose. First, in the short term I want to fix up the general __KERNEL__ vs !__KERNEL__ brokenness in include/linux and clean up and document which APIs are userspace ABI by moving them to include/kabi. If that first part works, then in the very _long_ term, (on the order of a couple years or so) I would like to encourage larger projects like klibc and GLIBC to migrate to using *only* the include/kabi headers to avoid namespace pollution and allow kernel devs to rearrange and adjust the linux/* headers as they see fit without caring about breaking external programs.
Once the libcs are using kabi/*, GLIBC can pick up a set of linux/*.h "emulation" files based on kabi/*.h that provide all of the old source-APIs for as long as anybody cares, and Linux can go back to screwing around with its header files in any way that it pleases (just as long as it keeps ABI stuff all in kabi). One of my major goals for this rework is to ensure that the kabi/*.h header files actually define the _ABI_ going back to the very early releases of Linux. As Linus said, it's an ABI, which means that adding new features is OK, but removing them is bad, so I'll try to make sure it's _perfectly_ _ok_ to use a 2.6 kabi/ on a 2.0 kernel. It should also be perfectly ok to use a 2.6 kabi/ on a 5-years-down-the-road 2.8 kernel as long as you restrict yourself to the subset of features that were available in 2.6.
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 4 lines · Kyle Moffett wrote/
Quoted 2 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Quoted 14 lines
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
1) The Linux kernel compiles and works at every step along the way
2) Since most of the headers are currently quite broken with respect to GLIBC and userspace, I won't spend much extra time preserving compatibility with GLIBC, userspace, or non-GCC compilers.
3) Everything ininclude/kabiwill have a__kabi_or__KABI_prefix.
4) Headers ininclude/linuxthat need the KABI interfaces will include the corresponding <kabi/*.h> header and define or typedef the necessary KABI definitions to the names the kernel wants.
5) The stuff ininclude/kabi/*.hshould always be completely independent of userspace/kernelspace and not require any includes outside of <kabi/*>. This means that the only preprocessor symbols that we can assume are present are those provided by the compiler itself.
Hello,
I completely agree with rules 1, 2 and 5. However, IMHO rule 4 should just be the inverse of rule 5: The stuff in include/linux should always be independent from KABI (and userspace of course). Simply because the way we _implement_ things in the kernel has to be different from the things that we _specify_ in the kernel ABI. They just append to be both written in C language, but it's not a reason to mix them. The kernel developers has to be free of doing any kludge, clever things, compatibility workarounds without affecting the userspace applications. Otherwise, you'll end up with another include/linux after few months! Separating the implementation and the binary specification has the additional advantage that if some kernel hacker mistakenly change the ABI, it's easy to say : "see, after your commit xxxxxxxx, the linux header and the kabi header are semantically different. You did something Wrong".
As for rule 3, if you have independent headers, this should be much less necessary. Additionally, keeping all the names identical to what they are already called will allow userspace to just use include/kabi/ as the /usr/include/linux/ directory. Avoiding smelly things like:
linux/foo.h:
#define __kabi_foo foo
#include <kabi/foo.h>
That was my 2 cents :-)
Regards,
Eric
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 5 lines · Eric Piel
I completely agree with rules 1, 2 and 5. However, IMHO rule 4 should just be the inverse of rule 5: The stuff in
include/linuxshould always be independent from KABI (and userspace of course). Simply because the way we _implement_ things in the kernel has to be different from the things that we _specify_ in the kernel ABI.
You know all the stuff that's marked __user? It's all kernel ABI. Having it defined in two places invites version skew, and the kernel needs it too because the kernel is parsing the stuff sent in by userspace and filling it out to send back to userspace.
Lots of syscalls and ioctls and such pass a structure back and forth from userspace to the kernel and back, right? Doing a stat() fills out a structure, doing an losetup fills out a structure, and so on.
Userspace needs to know what's in this structure. It may be wrapped in a libc function that fills out a different structure from the kernel structure, but the data that goes back and forth between the program and the kernel has to be defined in a header somewhere so the libc knows what the kernel's sending and the kernel knows what the libc is sending. (And for those functions with no libc wrapper, the user program needs to know the structure directly, somehow.)
Having a data-marshalling ABI structure defined in two places invites version skew. Userspace needs access to this (at least to build a libc), and the kernel needs access to this, because it's a _communication_mechanism_. You can't have a communication mechanism that's only defined at one end.
Quoted 8 lines
As for rule 3, if you have independent headers, this should be much less necessary. Additionally, keeping all the names identical to what they are already called will allow userspace to just use
include/kabi/as the/usr/include/linux/directory. Avoiding smelly things like:
linux/foo.h:#define __kabi_foo foo #include <kabi/foo.h>
The #define is, indeed, smelly.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 7 lines · Rob Landley wrote/
Quoted 5 lines · Eric Piel
I completely agree with rules 1, 2 and 5. However, IMHO rule 4 should just be the inverse of rule 5: The stuff in
include/linuxshould always be independent from KABI (and userspace of course). Simply because the way we _implement_ things in the kernel has to be different from the things that we _specify_ in the kernel ABI.
Quoted 7 lines
You know all the stuff that's marked
__user? It's all kernel ABI. Having it defined in two places invites version skew, and the kernel needs it too because the kernel is parsing the stuff sent in by userspace and filling it out to send back to userspace.Lots of syscalls and ioctls and such pass a structure back and forth from userspace to the kernel and back, right? Doing a
stat()fills out a structure, doing an losetup fills out a structure, and so on.Userspace needs to know what's in this structure. It may be wrapped in a libc function that fills out a different structure from the kernel structure, but the data that goes back and forth between the program and the kernel has to be defined in a header somewhere so the libc knows what the kernel's sending and the kernel knows what the libc is sending. (And for those functions with no libc wrapper, the user program needs to know the structure directly, somehow.)
Having a data-marshalling ABI structure defined in two places invites version skew. Userspace needs access to this (at least to build a libc), and the kernel needs access to this, because it's a _communication_mechanism_. You can't have a communication mechanism that's only defined at one end.
Well, that's half true. Indeed, in general, having two separate definitions invites version skew. However, in this particular case, it's slightly different: because the principle of the ABI is to be stable, or more exactly _compatible_. This means that if one definition was right at some point in the time, it should always still be true ten years later. At worse, the ABI can be extended, but never changed. If the specification (KABI) and the implementation (Linux) are not compatible it means the kernel developers screwed up, not that the KABI maintainers haven't updated in time.
Of course, next to the theory, there is the reality. Some part of the ABI has already be changed (broken) in the past (like the alsa ABI, IIRC). In such case the KABI maintainers will have to handle the changes as promptly as possible, but the responsibility will be held by the kernel developer who has opted for breakage.
The real problem of sharing the same headers between kernel and KABI is that it will end up by having to re-implement the "#ifdef __KERNEL__"'s. Have a look at Kyle's second patch "Generalize fd_set handling across architectures". Some headers had a different version of the __FD_*() macros depending on the compiler. That's something you may want to have in the implementation but definitely not in the specification. In this situation, Kyle handled it nicely by writing versions compatible with any compiler. Good looking solution inside the kernel will not always be an option. IMHO, if KABI and the kernel share the same headers, it's just a matter of time before someone introduces an "#ifdef __KERNEL__"-alike mechanism, exactly what we've been trying to remove.
Eric
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 1 line · Eric Piel
The real problem of sharing the same headers between kernel and KABI is that it will end up by having to re-implement the "#ifdef __KERNEL__"'s. Have a look at Kyle's second patch "Generalize
fd_sethandling across architectures". Some headers had a different version of the __FD_*() macros depending on the compiler. That's something you may want to have in the implementation but definitely not in the specification.
Actually, I think it's the other way around. The <kabi/*.h> files should (eventually) be useable in basically any compilation environment thrown at it. This means it should work from C and C++, using GCC, ICC, or some custom barely-standards-compliant compiler. I didn't bother with that part right now, since I still want to try to reuse the generic bitops if possible, but it's something I plan to address in future versions of the patchset (see below).
Quoted 1 line
In this situation, Kyle handled it nicely by writing versions compatible with any compiler.
Eh, not really. "__inline__" is GCC-specific and probably won't work in other compilers (unless you did "#define __inline__", which would bloat the code a lot).
This case highlights something else I'd like to do. A good chunk of the functionality in the Linux kernel works both in userspace and kernelspace, and some of those arch-specific primitives (like the inline bitops) would be useful in defining the kabi headers. According to Jeff Dike, UML would like access to some of that stuff unrestricted by __KERNEL__ too. In all of those cases, it's not an ABI and all the users are in-kernel so it could be changed at will. I'd like to try to put some of that into a "klib" directory (though with dependencies crossing between kabi and klib) so that it could be used in kabi and UML without duplicating functionality. Naturally much of that would be C-only and depend on GCC, but I would have to be careful that the kabi portions used least-common-denominator functionality.
That brings up one final point: Does anybody actually use any compilers on Linux other than GCC?
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 2 lines · Kyle Moffett
According to Jeff Dike, UML would like access to some of that stuff unrestricted by
__KERNEL__too.
Not quite true - I just want them physically separated from the non-userspace-usable stuff, as in different files. I don't care whether they are inside #ifdef __KERNEL__, because that's defined in the UML build.
Jeff
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 5 lines
Eh, not really. "
__inline__" is GCC-specific and probably won't work in other compilers (unless you did "#define__inline__", which would bloat the code a lot).
But ___inline is a C99 keyword, isnot it?
Jan Engelhardt
[OT] Non-GCC compilers used for linux userspace
Quoted 3 lines · Jan Engelhardt
Quoted 1 line
Eh, not really. "
__inline__" is GCC-specific and probably won't work in other compilers (unless you did "#define__inline__", which would bloat the code a lot).But
___inlineis a C99 keyword, is not it?
Not even GCC fully supports C99 (although I think it does support that keyword when passed -std=c99 or -std=gnu99), and I suspect that a majority of the other compilers for which we would want to add support in the kernel headers would not support C99 or would do a poor job of handling inline functions.
But my question still stands. Does anybody actually use any non-GCC compiler for userspace in Linux?
Cheers,
Kyle Moffett
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 2 lines · Kyle Moffett wrote/
But my question still stands. Does anybody actually use any non-GCC compiler for userspace in Linux?
At least in the domain of HPC, I've seen people which were compiling mostly *everything* with the intel compiler (x86 and ia64) for performance reason. So... yes userspace is sometimes compiled with non-GCC compiler :-)
Eric
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 4 lines · Eric Piel
Quoted 1 line · Kyle Moffett wrote/
But my question still stands. Does anybody actually use any non-GCC compiler for userspace in Linux?
At least in the domain of HPC, I've seen people which were compiling mostly *everything* with the intel compiler (x86 and ia64) for performance reason. So... yes userspace is sometimes compiled with non-GCC compiler :-)
Ok, well, the Intel compiler actually ends up emulating GCC and supports most of its extensions; supposedly it can even be used to compile the kernel sources, as per include/linux/compiler-intel.h. (I don't know how recently this has been tested, though) So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
Cheers,
Kyle Moffett
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"KM" == Kyle Moffett <[email protected]> writes:
KM> So does anybody compile userspace under anything other than GCC or KM> Intel compilers? Do any such compilers even exist?
PGI and PathScale are around. Lahey, too, although they seem to just do Fortran now.
I doubt you'd want to worry about compiling the entire userland with these compilers, however.
- J<
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 6 lines · Jason L Tibbitts III
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"KM" == Kyle Moffett <[email protected]> writes:
So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
PGI and PathScale are around. Lahey, too, although they seem to just do Fortran now.
I doubt you'd want to worry about compiling the entire userland with these compilers, however.
Mainly I want to know if I should even bother making the kabi headers compile with anything other than GCC. Judging from the apparently negligible number of users, it doesn't sound like something I should spend much or any time on, at least for now.
Cheers,
Kyle Moffett
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 8 lines · Kyle Moffett
Quoted 7 lines · Jason L Tibbitts III
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"KM" == Kyle Moffett <[email protected]> writes:
So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
PGI and PathScale are around. Lahey, too, although they seem to just do Fortran now.
Quoted 7 lines
Quoted 2 lines
I doubt you'd want to worry about compiling the entire userland with these compilers, however.
Mainly I want to know if I should even bother making the kabi headers compile with anything other than GCC. Judging from the apparently negligible number of users, it doesn't sound like something I should spend much or any time on, at least for now.
I'm not sure how you got to that conclusion. People have already named several non-GCC compilers that are used; and most of the users of commercial compilers won't be reading this list.
If you want glibc to ever include these things, they had better be portable C and work without GCC. Otherwise it's a non-starter. Only GCC may be used to build glibc, but it deliberately supports any conforming C compiler to build userspace code.
Signature · 2 lines
Daniel Jacobowitz CodeSourcery
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 7 lines · Daniel Jacobowitz
Quoted 6 lines · Kyle Moffett
Quoted 5 lines · Jason L Tibbitts III
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"KM" == Kyle Moffett <[email protected]> writes:
So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
PGI and PathScale are around. Lahey, too, although they seem to just do Fortran now.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I doubt you'd want to worry about compiling the entire userland with these compilers, however.
Mainly I want to know if I should even bother making the kabi headers compile with anything other than GCC. Judging from the apparently negligible number of users, it doesn't sound like something I should spend much or any time on, at least for now.
I'm not sure how you got to that conclusion. People have already named several non-GCC compilers that are used; and most of the users of commercial compilers won't be reading this list.
If you want glibc to ever include these things, they had better be portable C and work without GCC. Otherwise it's a non-starter. Only GCC may be used to build glibc, but it deliberately supports any conforming C compiler to build userspace code.
Ok, my email was a bit premature (I've gotten a couple other private emails about other compilers too) and if people see this as an issue then I'll try to make all the code C89-compliant from the start. I just didn't want to go writing a whole bunch of compatibility macros only to find out that they never got used.
Cheers,
Kyle Moffett
Re: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 2 lines · Kyle Moffett
Mainly I want to know if I should even bother making the kabi headers compile with anything other than GCC.
The PathScale compiler is gcc-compatible, so there should be no problems for us using gcc-isms in those headers.
<b
Re: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 3 lines · Kyle Moffett
don't know how recently this has been tested, though) So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
tcc
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O für einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
Re: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 10 lines · Kyle Moffett
Quoted 8 lines · Eric Piel
Quoted 2 lines · Kyle Moffett wrote/
But my question still stands. Does anybody actually use any non- GCC compiler for userspace in Linux?
At least in the domain of HPC, I've seen people which were compiling mostly *everything* with the intel compiler (x86 and ia64) for performance reason. So... yes userspace is sometimes compiled with non-GCC compiler :-)
Quoted 7 lines
Ok, well, the Intel compiler actually ends up emulating GCC and supports most of its extensions; supposedly it can even be used to compile the kernel sources, as per
include/linux/compiler-intel.h. (I don't know how recently this has been tested, though) So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
Other compilers do exist.
Over the years I've personally used a few to compile userspace apps for different projects (though never for compiling the kernel).
Some of the compilers I have personally used for userspace apps on Linux include ;
lcc - http://www.cs.princeton.edu/software/lcc/
tcc - http://fabrice.bellard.free.fr/tcc/
(plus gcc & icc ofcourse)
Others that I know of but have never used include :
sdcc - http://sdcc.sourceforge.net/
Compaq C for Linux - http://h30097.www3.hp.com/linux/compaq_c/index.html
Open Watcom - http://www.openwatcom.org/
vacpp - http://www-306.ibm.com/software/awdtools/vacpp/
XL C/C++ - http://www.absoft.com/Products/Compilers/C_C++/Linux/XLC/XLC.html
and I'm sure many more exist...
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Re: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 17 lines · Kyle Moffett
Quoted 8 lines · Eric Piel
Quoted 2 lines · Kyle Moffett wrote/
But my question still stands. Does anybody actually use any non- GCC compiler for userspace in Linux?
At least in the domain of HPC, I've seen people which were compiling mostly *everything* with the intel compiler (x86 and ia64) for performance reason. So... yes userspace is sometimes compiled with non-GCC compiler :-)
Ok, well, the Intel compiler actually ends up emulating GCC and supports most of its extensions; supposedly it can even be used to compile the kernel sources, as per
include/linux/compiler-intel.h. (I don't know how recently this has been tested, though) So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
Quoted 2 lines
Cheers,
Kyle Moffett
I play with Fabrice Bellard's Tiny C Compiler (http://www.tinycc.org), and hope to get a distro compiled with it someday, at least as a proof of concept.
That aims for full c99 and is already implementing a lot of gcc stuff too.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [OT] Non-GCC compilers used for linux userspace
On Tue, 28 Mar 2006, Rob Landley announced authoritatively:
Quoted 3 lines
I play with Fabrice Bellard's Tiny C Compiler (http://www.tinycc.org), and hope to get a distro compiled with it someday, at least as a proof of concept.
It's a nifty idea, a sort of uClibc of C compilers. Alas it's useless on almost all my systems because it's x86-only by design...
Quoted 1 line
That aims for full c99 and is already implementing a lot of gcc stuff too.
Good for it, as long as it doesn't go on to define __GNUC__ like icc did at one point (even though it doesn't implement all GCC extensions)... but Fabrice is sane so I doubt he'd do anything that loopy.
Signature · 3 lines
`Come now, you should know that whenever you plan the duration of your unplanned downtime, you should add in padding for random management freakouts.'
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 7 lines · Nix
On Tue, 28 Mar 2006, Rob Landley announced authoritatively:
Quoted 3 lines
I play with Fabrice Bellard's Tiny C Compiler (http://www.tinycc.org),
and hope to get a distro compiled with it someday, at least as a proof of concept.It's a nifty idea, a sort of uClibc of C compilers. Alas it's useless on almost all my systems because it's x86-only by design...
Actually according to the changelog version 0.9.21 grew support for ARM, and I believe it supports some other platforms too.
Notice that Fabrice Bellard (who wrote it) is also the guy behind qemu. My vague understanding of how things went is that at some point after doing tccboot, he was decoupled the C parser from the code generator in order to retarget tcc to produce code for other platforms, and got the idea of producing different front-ends too so it could input something other than C, namely machine code.
The result was qemu, which sort of compiles machine code to machine code dynamically, and which has taken up a large chunk of his time ever since. (The speed of tcc development has tailed off noticeably since, but he still spends a little time on it, and there are other developers...)
Quoted 7 lines
Quoted 2 lines
That aims for full c99 and is already implementing a lot of gcc stuff too.
Good for it, as long as it doesn't go on to define
__GNUC__like icc did at one point (even though it doesn't implement all GCC extensions)... but Fabrice is sane so I doubt he'd do anything that loopy.
That's more a header issue anyway. That's the uClibc developers problem. :)
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [OT] Non-GCC compilers used for linux userspace
On Wed, 29 Mar 2006, Rob Landley whispered secretively:
Quoted 2 lines
Actually according to the changelog version 0.9.21 grew support for ARM, and I believe it supports some other platforms too.
That's... impressive. Of course the more generality it grows the slower it must necessarily become, although it'll be a while until its IR is as bloated and thus as slow as GCC's... (GCC's only slow because of cache effects, really).
Quoted 4 lines
The result was qemu, which sort of compiles machine code to machine code dynamically, and which has taken up a large chunk of his time ever since. (The speed of tcc development has tailed off noticeably since, but he still spends a little time on it, and there are other developers...)
Well, I'd rather he spent time on qemu than tcc; there are other C compilers but there's nothing quite like qemu (bochs doesn't work very well, valgrind is similar in essence but very different in operation...)
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
That aims for full c99 and is already implementing a lot of gcc stuff too.
Good for it, as long as it doesn't go on to define
__GNUC__like icc did at one point (even though it doesn't implement all GCC extensions)... but Fabrice is sane so I doubt he'd do anything that loopy.That's more a header issue anyway. That's the uClibc developers problem. :)
;)
Signature · 3 lines
`Come now, you should know that whenever you plan the duration of your unplanned downtime, you should add in padding for random management freakouts.'
Re: [OT] Non-GCC compilers used for linux userspace
Quoted 6 lines · Nix
On Wed, 29 Mar 2006, Rob Landley whispered secretively:
Quoted 2 lines
Actually according to the changelog version 0.9.21 grew support for ARM, and I believe it supports some other platforms too.
That's... impressive. Of course the more generality it grows the slower it must necessarily become,
Not if compliation speed is the primary explicit design goal from day one, and they regression test with that in mind.
Keep in mind that the main use of tcc these days is to turn c into a scripting language. Just start your C file with
#!/usr/bin/tcc -run
And notice that #! is a preprocessor comment line as far as tcc is concerned. :)
Quoted 8 lines
Quoted 4 lines
The result was qemu, which sort of compiles machine code to machine code dynamically, and which has taken up a large chunk of his time ever since. (The speed of tcc development has tailed off noticeably since, but he still spends a little time on it, and there are other developers...)
Well, I'd rather he spent time on qemu than tcc; there are other C compilers but there's nothing quite like qemu (bochs doesn't work very well, valgrind is similar in essence but very different in operation...)
They're still sort of related. The sad part is that tcc is -><- this close to building an unmodified linux kernel, as tccboot demonstrates. But Fabrice seems to have gone "ooh, shiny!" off in another direction, for entirely understandable reasons... :)
I'm sure somebody will go take a whack at it sooner or later. (I haven't got time either.)
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [OT] Non-GCC compilers used for linux userspace
On Thu, 30 Mar 2006, Rob Landley suggested tentatively:
Quoted 10 lines · Nix
Quoted 6 lines
On Wed, 29 Mar 2006, Rob Landley whispered secretively:
Quoted 2 lines
Actually according to the changelog version 0.9.21 grew support for ARM, and I believe it supports some other platforms too.
That's... impressive. Of course the more generality it grows the slower it must necessarily become,
Not if compliation speed is the primary explicit design goal from day one, and they regression test with that in mind.
Aaah.
Quoted 4 lines
Keep in mind that the main use of tcc these days is to turn c into a scripting language. Just start your C file with
#!/usr/bin/tcc -run
I feel distinctly queasy. (I can't easily think of a less suitable language for scripting than C, either: perhaps COBOL...)
Quoted 2 lines
And notice that #! is a preprocessor comment line as far as tcc is concerned. :)
I noticed that, but I didn't think anyone actually *used* tcc for this.
Signature · 3 lines
`Come now, you should know that whenever you plan the duration of your unplanned downtime, you should add in padding for random management freakouts.'
Re: [OT] Non-GCC compilers used for linux userspace
Nix <[email protected]> (Thu, 30 Mar 2006 23:02:59 +0100):
Quoted 6 lines
Quoted 2 lines
And notice that #! is a preprocessor comment line as far as tcc is concerned. :)
I noticed that, but I didn't think anyone actually *used* tcc for this.
I did...
Signature · 1 line
Hilsen Harald.
Re: [OT] Non-GCC compilers used for linux userspace
In reply to NixQuoted 7 lines · Nix
Quoted 4 lines
Keep in mind that the main use of tcc these days is to turn c into a scripting language. Just start your C file with
#!/usr/bin/tcc -run
I feel distinctly queasy. (I can't easily think of a less suitable language for scripting than C, either: perhaps COBOL...)
No, the queasy bit is when you realize that you can stick library linkage directives on that command line and thus you can dynamically compile and run X apps.
They have examples of doing this. :)
Quoted 4 lines
Quoted 2 lines
And notice that #! is a preprocessor comment line as far as tcc is concerned. :)
I noticed that, but I didn't think anyone actually *used* tcc for this.
It's not exactly "what it's for", but it's up there.
Rob
Signature · 1 line
Never bet against the cheap plastic solution.
Re: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 2 lines · [email protected] (Kyle Moffett) writes:
So does anybody compile userspace under anything other than GCC or Intel compilers? Do any such compilers even exist?
ccc on alpha
Signature · 4 lines
Mathieu Chouquet-Stringer
"Le disparu, si l'on vénère sa mémoire, est plus présent et
plus puissant que le vivant".
-- Antoine de Saint-Exupéry, Citadelle --Re: [OT] Non-GCC compilers used for linux userspace
In reply to Eric PielQuoted 6 lines · Eric Piel
Quoted 2 lines · Kyle Moffett wrote/
But my question still stands. Does anybody actually use any non-GCC compiler for userspace in Linux?
At least in the domain of HPC, I've seen people which were compiling mostly *everything* with the intel compiler (x86 and ia64) for performance reason. So... yes userspace is sometimes compiled with non-GCC compiler :-)
The pathscale, and pgi compilers also get a reasonable amount of use.
pathscale has a gcc derived front-end so it isn't to much to worry about.
The pgi compiler as I recall is a fairly pedantic c90 compiler, that doesn't try and support gcc extensions.
Eric
Re: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Kyle" == Kyle Moffett <[email protected]> writes:
Kyle> But my question still stands. Does anybody actually use any Kyle> non-GCC compiler for userspace in Linux?
Yes. GCC produces much less than optimal code on Itanium, so the Intel compiler often gets used.
--
Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au ERTOS within National ICT Australia
RE: [OT] Non-GCC compilers used for linux userspace
In reply to Kyle MoffettQuoted 8 lines
Not even GCC fully supports C99 (although I think it does support that keyword when passed -std=c99 or -std=gnu99), and I suspect that a majority of the other compilers for which we would want to add support in the kernel headers would not support C99 or would do a poor job of handling inline functions.
Quoted 5 lines
But my question still stands. Does anybody actually use any non-GCC compiler for userspace in Linux?
Cheers,
Kyle Moffett
Sometimes.
I have customers that use Pathscale, Intel, and Portland groups compilers.
Not all of them use the C compilers (Some just use the F90 compilers), but a number of them do use the C compiler, though I would be that most of the applications are not going to care about doing much against anything but standard ANSI C calls.
Roger
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Jan EngelhardtQuoted 5 lines · Jan Engelhardt
Quoted 4 lines
Eh, not really. "
__inline__" is GCC-specific and probably won't work in other compilers (unless you did "#define__inline__", which would bloat the code a lot).But
___inlineis a C99 keyword, isnot it?
___inline is not, but inline is.
Signature · 4 lines
"Platonically Evil Monkey has been symbolically representing the darkest fears of humanity since the dawn of literature and religion, and I think I speak for everyone when I give it a sidelong glance of uneasy recognition this evening." --Scrymarch
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 2 lines · Kyle Moffett
That brings up one final point: Does anybody actually use any compilers on Linux other than GCC?
Yes, some people use (or plan to use) the ARM Ltd toolchain
(http://www.arm.com/products/DevTools/RVCT.html) to cross-compile applications for Linux (mainly for a better size/performance).
Signature · 1 line
Catalin
Arch-specific header inconsistency (asm-*/termios.h)
In reply to Kyle MoffettI'm in the process of cleaning up various headers for use from userspace, and at the same time I'm working to clean up some of the duplication in the kernel-only portions of those headers. I'm running into one oddity that I can't explain. In the various asm-*/termios.h files, there are user_termio_to_kernel_termios functions. The m68k header does this:
get_user((termios)->c_line, &(termio)->c_line);
The i386 header is missing that line, and after macro expansion the rest of the context is almost exactly identical. It appears there are a couple other architectures that fall on both sides of the fence. Can anyone explain this apparently aberrant behavior or is this a bug? If there's a reason behind it, I'll make sure to put in a useful comment; otherwise I'll fix it.
Thanks for the help!
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
In reply to Kyle MoffettQuoted 4 lines · Kyle Moffett
Quoted 2 lines · On Fri, 24 Mar 2006 17:46:27 -0500 Kyle Moffett <[email protected]> wrote:
I'm working on some sample patches now which I'll try to post in a few days if I get the time.
Quoted 14 lines
Ok, here's a sample of the KABI conversion and cleanup patches that I'm proposing. I have a few fundamental goals for these patches:
1) The Linux kernel compiles and works at every step along the way
2) Since most of the headers are currently quite broken with respect to GLIBC and userspace, I won't spend much extra time preserving compatibility with GLIBC, userspace, or non-GCC compilers.
3) Everything ininclude/kabiwill have a__kabi_or__KABI_prefix.
4) Headers ininclude/linuxthat need the KABI interfaces will include the corresponding <kabi/*.h> header and define or typedef the necessary KABI definitions to the names the kernel wants.
5) The stuff ininclude/kabi/*.hshould always be completely independent of userspace/kernelspace and not require any includes outside of <kabi/*>. This means that the only preprocessor symbols that we can assume are present are those provided by the compiler itself.
Hi Kyle.
Been watching this thread evolve for a while - and contributed a little myself.
But I fail to find a rationale for the selected approach.
There are at minimum the following options:
1) Go through current set of headers and sanitize them - using __KERNEL__ to identify kernel only stuff.
2) Keep user space interface (KABI in your term?) in include/ and slowly move kernel specific definitions somewhere else. This has the great advantage to keep backward compaitibility.
3) 'Preprocess' include/ and generate a set of KABI files based on current set of (cleaned up) kernel header files. 'Preporcess in '' because a C-preprocessor will not be sutilable.
4) Introduce a virgin KABI (your approach). The virgin KABI has no users today and does in no way preserve backward compatibility.
Can you try to explain why the virgin approach is superior to the others.
Sam
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 4 lines · Sam Ravnborg
Hi Kyle.
Been watching this thread evolve for a while - and contributed a little myself. But I fail to find a rationale for the selected approach.1) Go through current set of headers and sanitize them - using
__KERNEL__to identify kernel only stuff.
I'm working on this right now as step one. I see two big problems with this approach. Primarily it leaves no distinction or documentation for future maintenance; the sanitization doesn't last long at all, as we've seen by trying this approach in the past.
Quoted 1 line
2) Keep user space interface (KABI in your term?) in
include/and slowly move kernel specific definitions somewhere else. This has the great advantage to keep backward compaitibility.
The big problem with this approach is it starts off with the submission of a 1000-item change-the-world patchset to move all of the completely-kernel-only patches from linux/include to linux/kinclude; destroying all mergeability in the process and completely serializing development. Judging from the responses on this list when similar changes have happened before, that kind of thing is usually considered bad.
Quoted 1 line
3) 'Preprocess'
include/and generate a set of KABI files based on current set of (cleaned up) kernel header files. 'Preprocess in '' because a C-preprocessor will not be suitable.
I got the idea of preprocessing, but your sentence seems to have gotten mangled somewhere. Any chance you could clarify? From what I can see by looking through the current headers, preprocessing will not solve some of the duplication I'd like to try to clean up. One example being the duplication of bitops in various macros all over the place, FD_SET/FD_CLR/FD_ISSET being a prime example of that duplication. It would be really nice to be able to implement those in terms of __set_bit, etc, however those macros themselves must be made userspace-clean, including adding C89-compat macros for non-GCC compilers and other mild ugliness, even though they'd never be directly exposed for user programs.
Quoted 3 lines
4) Introduce a virgin KABI (your approach). The virgin KABI has no users today and does in no way preserve backward compatibility.
Can you try to explain why the virgin approach is superior to the others.
Your "#4" does not accurately describe what I feel I'm trying to do. In my first set of patches I'd like to create a set of KABI headers (recently I've been considering sticking them under "linux-klib" or similar) which provide a set of useful standards-clean compiler-independent typedefs, macros, and inline functions to both the kernel _and_ whatever ABI headers we decide to use. (working on that now) Those headers would _not_ be for use directly from userspace, but only from within the kernel and any in-kernel ABI headers. As a result, since they would be either shipped with the kernel which compiles against them or installed with the KABI headers which depend upon them, developers would be gleefully free from any obligations to maintain a stable interface within the linux-klib headers (although the code itself should be mostly C89 compliant).
As I do that I would like to adjust the kernel headers to use those klib includes, hopefully alleviating some of the unusual include-cycle problems in various headers (like the asm/posix_types.h wanting to use asm/bitops.h for FD_SET operations, which in some architectures indirectly includes asm/posix_types.h). Along with that process I would be fixing up __KERNEL__ wherever they seemed misplaced. Afterwards, I would try to clean up the actual guts of the ABI (like FD_SET, for example), I would migrate that to include/linux-kabi headers or similar, trying to contain all the C89-compat code and macros for non-GCC compilers in include/linux-{kabi,klib}
For future maintenance it would hopefully provide a clean and easily noticeable break between "You can modify include/linux as much as you damn well please" and "This patch touches include/linux-kabi, let's check it carefully for ABI breakage." I'd like to propose this as a natural extension to Greg KH's ABI documentation, sequestering the more sensitive code into a separate directory where it may be more carefully watched and documented. Feel free to continue to criticize and dissent; even if it's eventually determined that my idea won't work, hopefully some other useful ideas will come up in the discussion.
Thanks for the comments!
Cheers,
Kyle Moffett
Re: [RFC][PATCH 0/2] KABI example conversion and cleanup
Quoted 4 lines · Kyle Moffett
Quoted 3 lines
3) 'Preprocess'
include/and generate a set of KABI files based on current set of (cleaned up) kernel header files. 'Preprocess in '' because a C-preprocessor will not be suitable.
Quoted 11 lines
I got the idea of preprocessing, but your sentence seems to have gotten mangled somewhere. Any chance you could clarify? From what I can see by looking through the current headers, preprocessing will not solve some of the duplication I'd like to try to clean up. One example being the duplication of bitops in various macros all over the place, FD_SET/FD_CLR/FD_ISSET being a prime example of that duplication. It would be really nice to be able to implement those in terms of
__set_bit, etc, however those macros themselves must be made userspace-clean, including adding C89-compat macros for non-GCC compilers and other mild ugliness, even though they'd never be directly exposed for user programs.
These seem to be two completely independent problems. Reducing the amount of duplication is a good thing all by itself that can be done independent from finding a better way to provide the headers to user space.
About one year ago, I started down this route in order to autogenerate something like linux-libc-headers from the kernel tree, but never got around to submitting anything because I could not keep up with the changes.
What I did there was to put a Makefile into each include directory with a list of the files that should get installed (the list from llc) and then add a headers_install target to Makefile that would run each header file through an open-coded version of 'unifdef -U__KERNEL__ < include/$FILE > $PREFIX/include/$FILE'.
The advantage of this approach is that it minimises the amount of change to each of the current header files while making it relatively easy to audit by looking for wrong stuff (namespace pollution, CONFIG_* dependencies, ...) in the installed headers.
Arnd <><