Recent changes to sysctl.h breaks glibc
Hi
Some recent changes to include/linux/sysctl.h breaks glibc.
Problem is that __sysctl_args have been modified to use '__user', but that is only defined if __KERNEL__ is defined, because that is the only time compiler.h is included.
------------------------------------------------
--- linux-2.5.69-bk2/include/linux/sysctl.h 2003-05-05
01:53:31.000000000 +0200
+++ linux-2.5.69-bk12/include/linux/sysctl.h 2003-05-18
22:12:39.000000000 +0200
@@ -36,11 +36,11 @@
member of a struct __sysctl_args to have? */
struct __sysctl_args {
- int *name;
+ int __user *name;
int nlen;
- void *oldval;
- size_t *oldlenp;
- void *newval;
+ void __user *oldval;
+ size_t __user *oldlenp;
+ void __user *newval;
size_t newlen;
unsigned long __unused[4];
};
--------------------------------------------------
Question: Is this expected behaviour ?
Thanks,
Signature · 1 line
Martin Schlemmer
Quoted 4 lines · Martin Schlemmer
Some recent changes to
include/linux/sysctl.hbreaks glibc. Problem is that__sysctl_argshave been modified to use '__user', but that is only defined if__KERNEL__is defined, because that is the only timecompiler.his included.
Don't include the kernel headers in userspace.
-- wli
Quoted 8 lines · William Lee Irwin III
Quoted 4 lines · Martin Schlemmer
Some recent changes to
include/linux/sysctl.hbreaks glibc. Problem is that__sysctl_argshave been modified to use '__user', but that is only defined if__KERNEL__is defined, because that is the only timecompiler.his included.Don't include the kernel headers in userspace.
Yes, the standard answer. So what kernel headers should glibc be compiled against then ?
Signature · 1 line
Martin Schlemmer
Quoted 2 lines · Martin Schlemmer
Yes, the standard answer. So what kernel headers should glibc be compiled against then ?
None. But as glibc still hasn't been fixed use kernel headers from linux 2.4.
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Martin Schlemmer
Yes, the standard answer. So what kernel headers should glibc be compiled against then ?
None. But as glibc still hasn't been fixed use kernel headers from linux 2.4.
Right, so who are going to tell the glibc guys that ?
-----------------------------------------------------------------
configure: error: GNU libc requires kernel header files from Linux 2.0.10 or later to be installed before configuring. The kernel header files are found usually in /usr/include/asm and /usr/include/linux; make sure these directories use files from Linux 2.0.10 or later. This check uses <linux/version.h>, so make sure that file was built correctly when installing the kernel header
files. To use kernel headers not from /usr/include/linux, use the configure option --with-headers.
-----------------------------------------------------------------
I do not mind if thing break due to compatibility if needed to enhance an API, etc ... gcc does that enough.
The big problem however is that every time somebody screws up user space due to changes in kernel headers, the reply is always the same: "do not use kernel headers in user land"
Ok, lets say we stop doing that. How do anything user side find out specifics at compile time related to the kernel it should run on ? If we may not include kernel headers, what then?
Please understand, this is not me trolling, but everybody always have the same answer for issues like this, but no solutions.
Thanks,
Signature · 1 line
Martin Schlemmer
Quoted 4 lines · Martin Schlemmer
Right, so who are going to tell the glibc guys that ?
-----------------------------------------------------------------
configure: error: GNU libc requires kernel header files from Linux 2.0.10 or later to be installed before configuring.
Quoted 8 lines
The kernel header files are found usually in
/usr/include/asmand/usr/include/linux; make sure these directories use files from Linux 2.0.10 or later. This check uses <linux/version.h>, so make sure that file was built correctly when installing the kernel header
files. To use kernel headers not from/usr/include/linux, use the configure option --with-headers.
-----------------------------------------------------------------
IIRC you're supposed to use some sort of sanitized copy, not the things directly. IMHO the current state of affairs sucks as there is no standard set of ABI headers, but grabbing them right out of the kernel is definitely not the way to go.
-- wli
Quoted 5 lines · William Lee Irwin III
IIRC you're supposed to use some sort of sanitized copy, not the things directly. IMHO the current state of affairs sucks as there is no standard set of ABI headers, but grabbing them right out of the kernel is definitely not the way to go.
Ok, anybody know of an effort to get this done ?
Also, what about odd things that are more kernel dependant like imon support in fam for example ? The imon.h will not be in the 'sanitized copy' ....
Regards,
Signature · 1 line
Martin Schlemmer
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines
IIRC you're supposed to use some sort of sanitized copy, not the things directly. IMHO the current state of affairs sucks as there is no standard set of ABI headers, but grabbing them right out of the kernel is definitely not the way to go.
Quoted 4 lines · Martin Schlemmer
Ok, anybody know of an effort to get this done ? Also, what about odd things that are more kernel dependant like imon support in fam for example ? The
imon.hwill not be in the 'sanitized copy' ....
It probably has to be hand-crafted for cases like the above.
I'm unaware of anyone actually doing anything about this.
-- wli
Quoted 9 lines · Martin Schlemmer
Quoted 5 lines · William Lee Irwin III
IIRC you're supposed to use some sort of sanitized copy, not the things directly. IMHO the current state of affairs sucks as there is no standard set of ABI headers, but grabbing them right out of the kernel is definitely not the way to go.
Ok, anybody know of an effort to get this done ?
Red Hat Linux ships with a mostly sanitized header set for this.
Quoted 4 lines
Also, what about odd things that are more kernel dependant like imon support in fam for example ? The
imon.hwill not be in the 'sanitized copy' ....
apps that have such deep knowledge about internals are supposed to provide their own copy of the headers in RHL at least. But there are few of those.
Quoted 2 lines · Martin Schlemmer
like imon support in fam for example ? The
imon.hwill not be in the 'sanitized copy' ....
Imon is a braindead IRIX feature and SGI stopped doing Linux patches for it when dnotify became more common. So WTF are you talking about?
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Martin Schlemmer
like imon support in fam for example ? The
imon.hwill not be in the 'sanitized copy' ....Imon is a braindead IRIX feature and SGI stopped doing Linux patches for it when dnotify became more common. So WTF are you talking about?
It was just an example ffs.
Signature · 1 line
Martin Schlemmer
Followup to: <[email protected]> By author: William Lee Irwin III <[email protected]> In newsgroup: linux.dev.kernel
Quoted 6 lines
IIRC you're supposed to use some sort of sanitized copy, not the things directly. IMHO the current state of affairs sucks as there is no standard set of ABI headers, but grabbing them right out of the kernel is definitely not the way to go.
This "cure" sucks worse than the disease. Now you're putting it onto everyone who maintains userspace to do the same repetitive task of "sanitizing" this. Especially for things this trivial, this is a ridiculous concept.
For 2.7, getting real exportable ABI headers is so bloody necessary it's not even funny. However, for 2.5, breaking things randomly is not the way to go.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Followup to: <[email protected]> By author: William Lee Irwin III <[email protected]> In newsgroup: linux.dev.kernel
Quoted 4 lines
Quoted 4 lines
IIRC you're supposed to use some sort of sanitized copy, not the things directly. IMHO the current state of affairs sucks as there is no standard set of ABI headers, but grabbing them right out of the kernel is definitely not the way to go.
Quoted 7 lines · H. Peter Anvin
This "cure" sucks worse than the disease. Now you're putting it onto everyone who maintains userspace to do the same repetitive task of "sanitizing" this. Especially for things this trivial, this is a ridiculous concept.
For 2.7, getting real exportable ABI headers is so bloody necessary it's not even funny. However, for 2.5, breaking things randomly is not the way to go.
I would rather have real exportable ABI headers, yes. We don't have them and AFAIK sanitized copies are the current policy.
-- wli
Quoted 11 lines · William Lee Irwin III
Quoted 7 lines · H. Peter Anvin
This "cure" sucks worse than the disease. Now you're putting it onto everyone who maintains userspace to do the same repetitive task of "sanitizing" this. Especially for things this trivial, this is a ridiculous concept.
For 2.7, getting real exportable ABI headers is so bloody necessary it's not even funny. However, for 2.5, breaking things randomly is not the way to go.
Quoted 3 lines
I would rather have real exportable ABI headers, yes. We don't have them and AFAIK sanitized copies are the current policy.
Unfortunately "the current policy" is unrealistic, and repeating it doesn't make it any less so.
-hpa
Quoted 7 lines · H. Peter Anvin
Quoted 7 lines
Quoted 7 lines
This "cure" sucks worse than the disease. Now you're putting it onto everyone who maintains userspace to do the same repetitive task of "sanitizing" this. Especially for things this trivial, this is a ridiculous concept.
For 2.7, getting real exportable ABI headers is so bloody necessary it's not even funny. However, for 2.5, breaking things randomly is not the way to go.
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines
I would rather have real exportable ABI headers, yes. We don't have them and AFAIK sanitized copies are the current policy.
Quoted 2 lines · H. Peter Anvin
Unfortunately "the current policy" is unrealistic, and repeating it doesn't make it any less so.
No contest there; unfortunately unrealistic amounts of work seem to be required to get around the general state of affairs at times. =( Does it really have to be 2.7? It seems most of this would be header reorganization with no runtime impact on the kernel.
-- wli
Quoted 11 lines · William Lee Irwin III
Quoted 2 lines · H. Peter Anvin
Unfortunately "the current policy" is unrealistic, and repeating it doesn't make it any less so.
No contest there; unfortunately unrealistic amounts of work seem to be required to get around the general state of affairs at times. =( Does it really have to be 2.7? It seems most of this would be header reorganization with no runtime impact on the kernel.
It is; it doesn't have to be in 2.7 if someone would be willing to step up and do it in time for 2.6. Personally I wouldn't have a prayer; I'm already way to deep into too many projects...
-=hpa
Quoted 10 lines · H. Peter Anvin
Quoted 2 lines
Unfortunately "the current policy" is unrealistic, and repeating it doesn't make it any less so.
No contest there; unfortunately unrealistic amounts of work seem to be required to get around the general state of affairs at times. =( Does it really have to be 2.7? It seems most of this would be header reorganization with no runtime impact on the kernel.
Well, this is a tender issue - both for users and non kernel developers working on a project that may (according to the author at least *g*) need kernel headers.
Getting something done sooner than later would thus be appreciated from many venues, and will also get rid of your head aces =)
Trying to get back to my point ... I am sure there are many people that will help getting this done, if only somebody with enough knowledge of how the kernel devs want the ABI headers to look like will take some kind of lead in the effort - i2c and sensor support in 2.5 is a good example.
Regards,
Signature · 1 line
Martin Schlemmer
Quoted 3 lines · Martin Schlemmer
Ok, lets say we stop doing that. How do anything user side find out specifics at compile time related to the kernel it should run on ?
They don't. You can run the same userspace on a wide range of kernels. I'd just leave the job of selcting your headers to the distro vendor - if they are too stupid to get their headers sanitized I'd just use a different distro.
Quoted 10 lines · Christoph Hellwig
Quoted 3 lines · Martin Schlemmer
Ok, lets say we stop doing that. How do anything user side find out specifics at compile time related to the kernel it should run on ?
They don't. You can run the same userspace on a wide range of kernels. I'd just leave the job of selcting your headers to the distro vendor - if they are too stupid to get their headers sanitized I'd just use a different distro.
Ok, so say they use LFS ?
Point is just that people like you keep on bitching about not using sanitized kernel headers, but do nothing about it, or until today have said nothing about 'sanitized headers'.
Signature · 1 line
Martin Schlemmer
Quoted 3 lines · Martin Schlemmer
Point is just that people like you keep on bitching about not using sanitized kernel headers, but do nothing about it, or until today have said nothing about 'sanitized headers'.
Why don't you just get the glibc-kernheaders package from rawhide (or the equivalent from your prefered distribution) and stop complaining?
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Martin Schlemmer
Point is just that people like you keep on bitching about not using sanitized kernel headers, but do nothing about it, or until today have said nothing about 'sanitized headers'.
Why don't you just get the glibc-kernheaders package from rawhide (or the equivalent from your prefered distribution) and stop complaining?
Since this is brought up againg.
Today we have:
include/asm-<arch>include/linux
include/<subsystem>
Within include/linux grep told me that "#ifdef __KERNEL__" was present in 219 places. So a lot of header files are kept in a shape that allows them to be included from user-land.
I see two possible solutions here:
1) Automate the process of creating sanitized user-land headers.
- It should be a trivial task to do - but will require some effort to be done right. Does any tool exist to do it today? Automation allows glibc to user kernel headers for a recent kernel without looking up a distribution specific set of header files somewhere.
Dave M brougt up one issue the other day, when he added a new define to a header file. No need to wait for a distribution to pick up.
2) Create a user level hirachy under include/ [obviously 2.7 material]
include/linux/user
include/<subsystem>/user
include/asm-<arch>/user
In reality we will see a lot of files in include/linux that looks like:
sched.h:
part 1 - all relevant includes:
#include <linux/.....> ... part 2 - user level interface #include <linux/user/sched.h>
part 3 - kernel specific definitions.
extern rwlock_t tasklist_lock; extern spinlock_t mmlist_lock;
I am well aware that by default glibc shall NOT use kernel headers. But IMHO it is too difficult to upgrate to a new version of the kernel headers.
Care to explain what is wrong with the above approaches - I may have missed something obvious.
Sam
Quoted 4 lines · Sam Ravnborg
Within
include/linuxgrep told me that "#ifdef__KERNEL__" was present in 219 places. So a lot of header files are kept in a shape that allows them to be included from user-land.
No. They're definitely _not_ "kept in shape".
A number of headers have historical baggage, mainly to support the old libc5 habits, and because removing the ifdef's is something that nobody has felt was worth the pain.
I think the only header file that should be considered truly exported is something like "asm/posix_types.h". For the others, we'll add __KERNEL__ protection on demand if the glibc guys can give good arguments that it helps them do the "copy-and-cleanup" phase.
Linus
Followup to: <[email protected]> By author: Linus Torvalds <[email protected]> In newsgroup: linux.dev.kernel
Quoted 10 lines
A number of headers have historical baggage, mainly to support the old libc5 habits, and because removing the ifdef's is something that nobody has felt was worth the pain.
I think the only header file that should be considered truly exported is something like "
asm/posix_types.h". For the others, we'll add__KERNEL__protection on demand if the glibc guys can give good arguments that it helps them do the "copy-and-cleanup" phase.
Copy and cleanup isn't realistic either, though, because it doesn't track ABI changes. ABI headers is the only realistic solution. We can't realistically get real ABI headers for 2.5, so please don't just break things randomly until then.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Quoted 8 lines · H. Peter Anvin
Followup to: <[email protected]> By author: Linus Torvalds <[email protected]> In newsgroup: linux.dev.kernelQuoted 5 lines
A number of headers have historical baggage, mainly to support the old libc5 habits, and because removing the ifdef's is something that nobody has felt was worth the pain.
Quoted 8 lines
Quoted 5 lines
I think the only header file that should be considered truly exported is something like "
asm/posix_types.h". For the others, we'll add__KERNEL__protection on demand if the glibc guys can give good arguments that it helps them do the "copy-and-cleanup" phase.Copy and cleanup isn't realistic either, though, because it doesn't track ABI changes.
ABI changes or ABI additions?
If the ABI is not fixed that is a bug. Admittedly some interfaces in the development kernel are still under development and so have not stabilized on an ABI but that is a different issue.
Quoted 3 lines
ABI headers is the only realistic solution. We
can't realistically get real ABI headers for 2.5, so please don't just break things randomly until then.
As the ABI remains fixed I remain unconvinced. Multiple implementations against the same ABI should be possible. The real question which is the more scalable way to do the code.
What I find truly puzzling is that after years glibc still needs kernel headers at all.
Eric
Quoted 7 lines · Eric W. Biederman
ABI changes or ABI additions?
If the ABI is not fixed that is a bug. Admittedly some interfaces in the development kernel are still under development and so have not stabilized on an ABI but that is a different issue.
ABI fixes and ABI additions, as well as outright ABI changes (yes they suck, but they happen.)
Quoted 8 lines
Quoted 4 lines
ABI headers is the only realistic solution. We
can't realistically get real ABI headers for 2.5, so please don't just break things randomly until then.As the ABI remains fixed I remain unconvinced. Multiple implementations against the same ABI should be possible. The real question which is the more scalable way to do the code.
The ABI doesn't remain fixed. Like everything else it evolves.
Quoted 2 lines
What I find truly puzzling is that after years glibc still needs kernel headers at all.
What I find truly puzzling is that obviously intelligent people like yourself still seem to think that ABIs remain fixed.
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 7 lines · Eric W. Biederman
ABI changes or ABI additions?
If the ABI is not fixed that is a bug. Admittedly some interfaces in the development kernel are still under development and so have not stabilized on an ABI but that is a different issue.
ABI fixes and ABI additions, as well as outright ABI changes (yes they suck, but they happen.)
And ABI changes from the ABI exposed by a stable kernel are _B_U_G_S. Yes _b_u_g_s happen but they are still _b_u_g_s.
It is legal to stop supporting an interface but changing the interface in such a way that you must know the version of the kernel you are running on is a _b_u_g.
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
ABI headers is the only realistic solution. We
can't realistically get real ABI headers for 2.5, so please don't just break things randomly until then.As the ABI remains fixed I remain unconvinced. Multiple implementations against the same ABI should be possible. The real question which is the more scalable way to do the code.
The ABI doesn't remain fixed. Like everything else it evolves.
I meant to say the existing subset of the ABI, remains fixed. Evolution by accretion is fine.
Quoted 5 lines
Quoted 2 lines
What I find truly puzzling is that after years glibc still needs kernel headers at all.
What I find truly puzzling is that obviously intelligent people like yourself still seem to think that ABIs remain fixed.
Simply because if the existing subset of an ABI does not remain fixed it is a _b_u_g. Only once in a blue moon is there a chance for an incompatible change, like when switching to x86-64.
And given my experiences running old a.out binaries Linux does a pretty good job at this. And I am not willing to admit that changes that break backwards compatibility (except for applications that depended on implementation bugs) are anything except bugs. That would only encourage changes that should not happen.
I do agree however that the ABI should be better documented. And that being able to automatically generate headers from the official documentation would be advantageous. With good documentation it should actually be harder to change an ABI because what the old ABI was will be clearer.
And I do agree that if the kernel builds with headers automatically generated from the official documentation. Then it will be easy to guarantee they are in sync.
But there is no reason not to write documentation today about what the kernel interfaces are and convert glibc and the kernel later when it is convenient to their development cycles.
What I do not is see the necessity of using automation to follow the documentation.
Eric
Quoted 11 lines · Eric W. Biederman
Quoted 3 lines
ABI fixes and ABI additions, as well as outright ABI changes (yes they suck, but they happen.)
And ABI changes from the ABI exposed by a stable kernel are _B_U_G_S. Yes _b_u_g_s happen but they are still _b_u_g_s.
It is legal to stop supporting an interface but changing the interface in such a way that you must know the version of the kernel you are running on is a _b_u_g.
Yes. And guess what? Bugs happen. Sometimes you can't fix them either because the "new" usage has gotten established. However, that's not the point.
Your message assumes that the ABI remains fixed. This is totally and utterly and undeniably WRONG. There are rules for how it may evolve, but it very much does evolve. No amount of handwaving or putting underscores in weird places will change that.
Quoted 8 lines
But there is no reason not to write documentation today about what the kernel interfaces are and convert glibc and the kernel later when it is convenient to their development cycles.
What I do not is see the necessity of using automation to follow the documentation.
Otherwise you have three places to manually make your changes (usually more) -- the documentation, the kernel, and glibc... and really you also have klibc, uclibc, dietlibc, and God knows what else.
Automation is the way to maintain these together and in concert, to avoid your "B_ U_ G_ S_." This isn't just a Good Thing, this is the only sane possibility.
Does that mean C source code is the only possible format? It most certainly *doesn't* -- in fact one could argue it's not even a very good format -- as long as C source code is one of the possible productions.
-hpa
Quoted 6 lines · H. Peter Anvin
Yes. And guess what? Bugs happen. Sometimes you can't fix them either because the "new" usage has gotten established. However, that's not the point.
Your message assumes that the ABI remains fixed. This is totally and utterly and undeniably WRONG.
It is correct by definition, the existing part of an ABI may not change.
Quoted 3 lines
There are rules for how it may evolve,
but it very much does evolve. No amount of handwaving or putting underscores in weird places will change that.
Sure you can add new functionality in a defined way. You can add to the ABI. You may not modify an existing definition. But that has no implications for existing working code.
Quoted 11 lines
Quoted 7 lines
But there is no reason not to write documentation today about what the kernel interfaces are and convert glibc and the kernel later when it is convenient to their development cycles.
What I do not is see the necessity of using automation to follow the documentation.
Otherwise you have three places to manually make your changes (usually more) -- the documentation, the kernel, and glibc... and really you also have klibc, uclibc, dietlibc, and God knows what else.
But since the older interface still remains you don't have to change klibc, uclibc, dietlibc, and whatever else does not care. Making it easy to change an existing ABI is simply wrong.
If you were to increment the syscall numbers by one no amount of automation in the world would make that a kernel that would be usable on a production box. The only way it could even work is if you were to declare that kernel uses a new ABI.
Quoted 3 lines
Automation is the way to maintain these together and in concert, to avoid your "B_ U_ G_ S_." This isn't just a Good Thing, this is the only sane possibility.
If things must be maintained in concert it is a bug.
With a fixed ABI people take advantage of new features as they care for them. And in general to use new features requires new code.
If people are adding and changing ioctls/sysctls/prctls left and right, and that is what is causing the maintenance problem, then that is the problem. And that is where the problem needs to be reigned in at.
Quoted 3 lines
Does that mean C source code is the only possible format? It most certainly *doesn't* -- in fact one could argue it's not even a very good format -- as long as C source code is one of the possible productions.
Agreed.
Calling it documentation simply makes it clear what the headers are, and suggest that the machine readable for does not need to be C source code.
Eric
Quoted 4 lines · Eric W. Biederman
If things must be maintained in concert it is a bug.
With a fixed ABI people take advantage of new features as they care for them. And in general to use new features requires new code.
And if the kernel headers aren't maintained in concert with the kernel, new userspace code can't reach the new features.
Therefor, by your definition, the current situation is a bug.
Try compiling code that uses futexes on a system that has a kernel that supports them, but kernel-headers that haven't been upgraded to mention them. The kernel has the new code, the userspace has the new code, but gcc will turn around and whinge about the new code because there's a piece missing in between. So people *CANT* take advantage of the new features (unless they do something silly like drag their own foo.h file around where it can get out of sync with reality).
Quoted 10 lines · [email protected] writes:
Quoted 4 lines · Eric W. Biederman
If things must be maintained in concert it is a bug.
With a fixed ABI people take advantage of new features as they care for them. And in general to use new features requires new code.
And if the kernel headers aren't maintained in concert with the kernel, new userspace code can't reach the new features.
Therefor, by your definition, the current situation is a bug.
Yes, glibc uses kernel headers.
Quoted 7 lines
Try compiling code that uses futexes on a system that has a kernel that supports them, but kernel-headers that haven't been upgraded to mention them. The kernel has the new code, the userspace has the new code, but gcc will turn around and whinge about the new code because there's a piece missing in between. So people *CANT* take advantage of the new features (unless they do something silly like drag their own
foo.hfile around where it can get out of sync with reality).
Or the build against a library that does that. There are not that many libraries.
For a lot of system calls it is actively dangerous to assume dev_t == __kernel_dev_t. As glibc does some cute things in there.
Eric
Quoted 2 lines · Eric W. Biederman
Or the build against a library that does that. There are not that many libraries.
So I get around the lack of foo-gram support in glibc by writing/compiling/installing a -lfoo-gram, which of course gets its header definitions from the same header file that kernheaders isn't supplying, or by dragging around the same local foo-gram.h that I'd otherwise be dragging around with my app.
Somehow, this doesn't seem like progress to me. And of course, it just sets me up for *odd* debugging problems when my foo-gram.h is at version 0.17 and the kernel and glibc are up to 0.25, and features mysteriously misbehave depending on the exact -I include order on the system I'm doing the build on, and whether the particular version of gcc has the automagic auto-ambush handling of /usr/local/include, and the phase of the moon...
Yes, I know I can build against a library. Point is that saying I can do it doesn't change the fact it's a hack and an ongoing maintenance problem.
Quoted 2 lines
For a lot of system calls it is actively dangerous to assume
dev_t==__kernel_dev_t. As glibc does some cute things in there.
I thought that sort of fun and games was *WHY* userspace can't use the kernel headers in the first place?
Followup to: <[email protected]> By author: [email protected] In newsgroup: linux.dev.kernel
Quoted 7 lines
Quoted 3 lines
For a lot of system calls it is actively dangerous to assume
dev_t==__kernel_dev_t. As glibc does some cute things in there.I thought that sort of fun and games was *WHY* userspace can't use the kernel headers in the first place?
Indeed. Because they try to export dev_t, not __kernel_dev_t (unless you know exactly what you're doing, which most people don't.)
dev_t should be defined by the library ABI, not by the kernel ABI.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · H. Peter Anvin
Yes. And guess what? Bugs happen. Sometimes you can't fix them either because the "new" usage has gotten established. However, that's not the point.
Your message assumes that the ABI remains fixed. This is totally and utterly and undeniably WRONG.
It is correct by definition, the existing part of an ABI may not change.
"THE ABI" does not consitute "THE EXISTING PART OF AN ABI". They are two different things.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
But there is no reason not to write documentation today about what the kernel interfaces are and convert glibc and the kernel later when it is convenient to their development cycles.
What I do not is see the necessity of using automation to follow the documentation.
Quoted 7 lines
Quoted 3 lines
Otherwise you have three places to manually make your changes (usually more) -- the documentation, the kernel, and glibc... and really you also have klibc, uclibc, dietlibc, and God knows what else.
But since the older interface still remains you don't have to change klibc, uclibc, dietlibc, and whatever else does not care. Making it easy to change an existing ABI is simply wrong.
No, it's not. It provides a chance to fix bugs quickly and easily. Furthermore, it provides a clean, uniform way to distribute ABI extensions.
Quoted 4 lines
If you were to increment the syscall numbers by one no amount of automation in the world would make that a kernel that would be usable on a production box. The only way it could even work is if you were to declare that kernel uses a new ABI.
Syscall numbers are an extremely bad example (ioctl structures is more like it), but since you brought it up: how would you deal with the hypotetical case that two syscalls were assigned the same number? You're screwed no matter what, and making it hard to fix doesn't improve the situation in any way.
Quoted 5 lines
Quoted 3 lines
Automation is the way to maintain these together and in concert, to avoid your "B_ U_ G_ S_." This isn't just a Good Thing, this is the only sane possibility.
If things must be maintained in concert it is a bug.
Things *DO* have to be maintained in concert if you want to get things done. I'm not talking atomic changes as I'm talking getting changes in within a reasonable time frame.
Quoted 6 lines
With a fixed ABI people take advantage of new features as they care for them. And in general to use new features requires new code.
If people are adding and changing ioctls/sysctls/prctls left and right, and that is what is causing the maintenance problem, then that is the problem. And that is where the problem needs to be reigned in at.
And your proposal for this is...?
Let's face it, you're going to need these kinds of things. You're also going to need a way to push these features into use. You're also going to want to have an automated way to make things like strace (my personal favourite debugging tool) produce useful output.
Quoted 9 lines
Quoted 3 lines
Does that mean C source code is the only possible format? It most certainly *doesn't* -- in fact one could argue it's not even a very good format -- as long as C source code is one of the possible productions.
Agreed.
Calling it documentation simply makes it clear what the headers are, and suggest that the machine readable for does not need to be C source code.
What it really is is an ABI specification, not documentation. The difference is that the specification *is* the ABI, by definition, whereas documentation is a *description* of what the ABI is. The latter may or may not be up to date and correct.
Automation prevents bugs. This is a good thing.
-hpa
Quoted 4 lines · H. Peter Anvin
Your message assumes that the ABI remains fixed. This is totally and utterly and undeniably WRONG. There are rules for how it may evolve, but it very much does evolve. No amount of handwaving or putting underscores in weird places will change that.
To a large extent, however, it merely grows. And in a lot of cases when it grows due to new syscalls, new interfaces, etc., you have to add matching code to glibc to use them _anyway_, so it's no problem for glibc's version of the headers to lag behind until the appropriate support is added.
You are, however, correct that the correct fix is to have completely separate headers which define the ABI. Then the real kernel headers in include/linux and include/asm can include them, and C libraries can also use them without contamination.
This requires that someone sit down and cut'n'paste large amounts of structures and definitions from include/linux/*.h into the new header files. I've been tempted to do that on occasion but what's held me up has been the fact that there isn't yet a consensus on how it should be laid out.
For compatibility with older libc, one approach would be to add a new directory to the include path which matches the existing layout (linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next from the actual kernel headers to pull in those files.
Alternatively, we could go further and take the opportunity to rearrange stuff further; I'm not sure what we really gain from that though other than extra pain.
If Linus would approve a strategy for rearranging the headers such that people can work on it without suspecting that they're just wasting their time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait accomplis -- if it isn't quite right, you end up having to do the whole thing from scratch, afaict.
Signature · 3 lines
dwmw2 ¹ but admittedly not all.
Quoted 7 lines · David Woodhouse
To a large extent, however, it merely grows. And in a lot of cases when it grows due to new syscalls, new interfaces, etc., you have to add matching code to glibc to use them _anyway_, so it's no problem for glibc's version of the headers to lag behind until the appropriate support is added.
... unless you need the new feature, may it be an ioctl to support your device driver, or whatnot.
Most ABI changes do not require
Quoted 10 lines
You are, however, correct that the correct fix is to have completely separate headers which define the ABI. Then the real kernel headers in
include/linuxandinclude/asmcan include them, and C libraries can also use them without contamination.This requires that someone sit down and cut'n'paste large amounts of structures and definitions from
include/linux/*.hinto the new header files. I've been tempted to do that on occasion but what's held me up has been the fact that there isn't yet a consensus on how it should be laid out.
I maintain the proposal I have given before:
<linux/abi/*.h> as the header file namespace;
<linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the migration is complete these are owned by the libc)
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
Some form of export of the expected syscall ABI as well as syscall numbering.
Quoted 8 lines
For compatibility with older libc, one approach would be to add a new directory to the include path which matches the existing layout (linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next from the actual kernel headers to pull in those files.
Alternatively, we could go further and take the opportunity to rearrange stuff further; I'm not sure what we really gain from that though other than extra pain.
I don't think the <linux/*.h> namespace as its currently laid out is very functional for exporting ABIs, so I'd like to
A bigger issue is if this really should be done in C. A worthwhile
thought: if this is done correctly then most or all of the 64/32 compat code (or any other arch1-on-arch2 compatibility) should be able to be automatically generated. If not, it almost certainly isn't done correctly...
Quoted 7 lines
If Linus would approve a strategy for rearranging the headers such that people can work on it without suspecting that they're just wasting their time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait accomplis -- if it isn't quite right, you end up having to do the whole thing from scratch, afaict.
Agreed.
-hpa
Quoted 1 line · H. Peter Anvin
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
What, do you mean _every_ kernel internal type would use such names? That's completely horrid...
-Miles
Signature · 2 lines
Come now, if we were really planning to harm you, would we be waiting here, beside the path, in the very darkest part of the forest?
Quoted 8 lines · Miles Bader
Quoted 1 line · H. Peter Anvin
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
What, do you mean _every_ kernel internal type would use such names? That's completely horrid...
No, I don't mean that.
The kernel-internal headers would typedef these to different names, e.g.
/* linux/types.h */
#include <linux/abi/types.h>
/* Kernel internal types */
typedef __kernel_dev64_t dev_t;
typedef __kernel_ino_t ino_t;
... etc ...
-hpa
Quoted 7 lines · H. Peter Anvin
The kernel-internal headers would typedef these to different names, e.g.
/*
linux/types.h*/
#include <linux/abi/types.h>
/* Kernel internal types */
typedef__kernel_dev64_tdev_t;
typedef__kernel_ino_tino_t;
I see... I guess that seems reasonable.
-Miles
Signature · 3 lines
`Cars give people wonderful freedom and increase their opportunities. But they also destroy the environment, to an extent so drastic that they kill all social life' (from _A Pattern Language_)
Quoted 9 lines · H. Peter Anvin
Quoted 7 lines · David Woodhouse
To a large extent, however, it merely grows. And in a lot of cases when it grows due to new syscalls, new interfaces, etc., you have to add matching code to glibc to use them _anyway_, so it's no problem for glibc's version of the headers to lag behind until the appropriate support is added.
Quoted 4 lines
... unless you need the new feature, may it be an ioctl to support your device driver, or whatnot.
Most ABI changes do not require
Ugh. I now see both sides of this.
ABI changes always require writing new code to take advantage of it but that code does not need to live in libc. And libc is expected to have the definitions necessary to compile the new code.
Quoted 11 lines
Quoted 10 lines
You are, however, correct that the correct fix is to have completely separate headers which define the ABI. Then the real kernel headers in
include/linuxandinclude/asmcan include them, and C libraries can also use them without contamination.This requires that someone sit down and cut'n'paste large amounts of structures and definitions from
include/linux/*.hinto the new header files. I've been tempted to do that on occasion but what's held me up has been the fact that there isn't yet a consensus on how it should be laid out.
Quoted 3 lines
I maintain the proposal I have given before:
<linux/abi/*.h> as the header file namespace;
What about <linux-abi/*.h>
Quoted 2 lines
<linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the migration is complete these are owned by the libc)
I think removing the abi namespace totally from the legacy directories helps this to a small extent.
Quoted 4 lines
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
Some form of export of the expected syscall ABI as well as syscall numbering.
Prototypes for everything including ioctls.
Quoted 9 lines
Quoted 8 lines
For compatibility with older libc, one approach would be to add a new directory to the include path which matches the existing layout (linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next from the actual kernel headers to pull in those files.
Alternatively, we could go further and take the opportunity to rearrange stuff further; I'm not sure what we really gain from that though other than extra pain.
Quoted 8 lines
I don't think the <linux/*.h> namespace as its currently laid out is very functional for exporting ABIs, so I'd like to
A bigger issue is if this really should be done in C. A worthwhile
thought: if this is done correctly then most or all of the 64/32 compat code (or any other arch1-on-arch2 compatibility) should be able to be automatically generated. If not, it almost certainly isn't done correctly...
Potentially. Certain things like type conversions may be a non-trivial exercise. It is certainly true that it should be possible to build the entire decoding logic for strace. Even in the mixed architecture cases.
For cases that are hard to automate see: sys_old_mmap vs. sys_mmap2, on x86. Potentially it can be done but it requires a powerful stub generator.
Quoted 9 lines
Quoted 7 lines
If Linus would approve a strategy for rearranging the headers such that people can work on it without suspecting that they're just wasting their time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait accomplis -- if it isn't quite right, you end up having to do the whole thing from scratch, afaict.
Agreed.
If it is done purely as headers certainly. For a header generator it might be something you can get away with. Because the core repository would not need to change just the world useable form. This may be the one good argument for doing this in something other than C.
Eric
Quoted 7 lines · Eric W. Biederman
Ugh. I now see both sides of this.
ABI changes always require writing new code to take advantage of it but that code does not need to live in libc. And libc is expected to have the definitions necessary to compile the new code.
Bingo.
Quoted 5 lines
Quoted 4 lines
I maintain the proposal I have given before:
<linux/abi/*.h> as the header file namespace;
Quoted 8 lines
What about <linux-abi/*.h>
Quoted 2 lines
<linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the migration is complete these are owned by the libc)
I think removing the abi namespace totally from the legacy directories helps this to a small extent.
It might, but the <linux/abi/*.h> namespace hasn't been used, so it isn't really legacy in the sense that you can have a possible conflict.
Having said that, <linuxabi/*.h> is the least bleacherous nonlegacy
alternative I've seen (I'd like to skip the dash, though.) <abi/*.h> is already taken.
Quoted 9 lines
Quoted 5 lines
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
Some form of export of the expected syscall ABI as well as syscall numbering.
Prototypes for everything including ioctls.
Absolutely. As I said in another post, if this is done correctly, a whole bunch of the ABI compatbility stuff like the 32-on-64 things should be possible to generate automatically. If that isn't true, it's done wrong.
Quoted 8 lines
Quoted 6 lines
A bigger issue is if this really should be done in C. A worthwhile
thought: if this is done correctly then most or all of the 64/32 compat code (or any other arch1-on-arch2 compatibility) should be able to be automatically generated. If not, it almost certainly isn't done correctly...
Quoted 9 lines
Potentially. Certain things like type conversions may be a non-trivial exercise. It is certainly true that it should be possible to build the entire decoding logic for strace. Even in the mixed architecture cases.
For cases that are hard to automate see:
sys_old_mmapvs.sys_mmap2, on x86. Potentially it can be done but it requires a powerful stub generator.
That one is somewhat tricky, and probably needs some ad hoc code. However, you get a long way toward the goal if you can express the following:
void * sys_mmap(void *, size_t, int, struct { int, int, off32_t }); void * sys_mmap2(void *, size_t, int, int, int, off32_t); void * sys_mmap64(void *, size_t, int, int, int, struct { off64_t });
(The latter doesn't exist at the moment AFAIK, but at some point it'll be necessary.)
You can get even further if you can express the fact that the final structure in sys_old_mmap are "memory arguments" -- ordinary arguments spilled to memory because we ran out of registers ("prematurely" in the case of sys_mmap, due to backwards compatibility constraints.)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
If Linus would approve a strategy for rearranging the headers such that people can work on it without suspecting that they're just wasting their time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait accomplis -- if it isn't quite right, you end up having to do the whole thing from scratch, afaict.
Quoted 7 lines
Quoted 1 line
Agreed.
If it is done purely as headers certainly. For a header generator it might be something you can get away with. Because the core repository would not need to change just the world useable form. This may be the one good argument for doing this in something other than C.
On the other hand, Linus just released "sparse", which is a pretty nice
C parser that might be useful for this, especially with some kind of
custom annotation capability. I talked to him about this a while ago, actually.
-hpa
Quoted 8 lines · H. Peter Anvin
Quoted 5 lines · Eric W. Biederman
Ugh. I now see both sides of this.
ABI changes always require writing new code to take advantage of it but that code does not need to live in libc. And libc is expected to have the definitions necessary to compile the new code.Bingo.
Several more little more points.
In general this only affects applications that are tied tightly to the kernel.
What the applications want is a kernel abi specification repository, which glibc is not, at least in a timely fashion, and it is not the emphasis of any libc so it is quite likely to be neglected.
So for applications today that cannot wait for the code to be incorporated into glibc the only reasonable advice is to include their own header files (that do not conflict with libc header files), for the ioctls and definitions they need. This is a maintainable if not beautiful strategy.
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
I maintain the proposal I have given before:
<linux/abi/*.h> as the header file namespace;
What about <linux-abi/*.h>
Quoted 2 lines
<linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the migration is complete these are owned by the libc)
I think removing the abi namespace totally from the legacy directories helps this to a small extent.
Quoted 5 lines
It might, but the <linux/abi/*.h> namespace hasn't been used, so it isn't really
legacy in the sense that you can have a possible conflict. Having said that, <linuxabi/*.h> is the least bleacherous nonlegacy alternative I've seen (I'd like to skip the dash, though.) <abi/*.h> is already taken.
No problem with skipping the dash. In general I would like to be able
to: ``rm -rf /usr/include/linux/ /usr/include/asm/'' and then compile
applications. This is not something I would recommend in a
production system any time soon but it would help in cruft detection.
Quoted 26 lines
Quoted 6 lines
Quoted 4 lines
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
Some form of export of the expected syscall ABI as well as syscall numbering.
Prototypes for everything including ioctls.
Absolutely. As I said in another post, if this is done correctly, a whole bunch of the ABI compatbility stuff like the 32-on-64 things should be possible to generate automatically. If that isn't true, it's done wrong.
Quoted 6 lines
Quoted 6 lines
A bigger issue is if this really should be done in C. A worthwhile
thought: if this is done correctly then most or all of the 64/32 compat code (or any other arch1-on-arch2 compatibility) should be able to be automatically generated. If not, it almost certainly isn't done correctly...Quoted 8 lines
Potentially. Certain things like type conversions may be a non-trivial exercise. It is certainly true that it should be possible to build the entire decoding logic for strace. Even in the mixed architecture cases.
For cases that are hard to automate see:sys_old_mmapvs.sys_mmap2, on x86. Potentially it can be done but it requires a powerful stub generator.
Quoted 2 lines
That one is somewhat tricky, and probably needs some ad hoc code. However, you get a long way toward the goal if you can express the following:
My basic point with respect to arch-on-arch compatibility is that sys_mmap -> sys_mmap2 (where they both take an off32_t argument) is classic case of the arch on arch compatibility.
The case is actually more interesting because sys_mmap2 uses a 4K aligned offset. Instead of a byte aligned offset. Which makes sys_mmap64 unnecessary for the time being.
But I am certainly for it where the work can be done easily.
A more interesting possibility to me at least is network transparency.
Quoted 15 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
If Linus would approve a strategy for rearranging the headers such that people can work on it without suspecting that they're just wasting their time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait accomplis -- if it isn't quite right, you end up having to do the whole thing from scratch, afaict.
Quoted 6 lines
Quoted 1 line
Agreed.
If it is done purely as headers certainly. For a header generator it might be something you can get away with. Because the core repository would not need to change just the world useable form. This may be the one good argument for doing this in something other than C.
Quoted 3 lines
On the other hand, Linus just released "sparse", which is a pretty nice C parser that might be useful for this, especially with some kind of custom annotation capability. I talked to him about this a while ago, actually.
Against doing this in C there are 3 arguments.
1) C may not be precise enough.
2) C may not contain enough information.
3) Reorganization by an automated tool may not be possible.
I don't know for certain that the ability to reorganize the generated interfaces from original source is important. A good design may make this a non-issue. But it may help prototyping so I mention it.
I suspect a good implementation of this would resemble to the CORBA IDL. A relative of C but with different restrictions.
Another very quite piece of the puzzle is how to include in the in kernel DSO.
Eric
Hi David.
Quoted 13 lines
Quoted 5 lines
Your message assumes that the ABI remains fixed. This is totally and utterly and undeniably WRONG. There are rules for how it may evolve, but it very much does evolve. No amount of hand-waving or putting underscores in weird places will change that.
To a large extent, however, it merely grows. And in a lot of cases when it grows due to new syscalls, new interfaces, etc., you have to add matching code to glibc to use them _anyway_, so it's no problem for glibc's version of the headers to lag behind until the appropriate support is added.
Quoted 5 lines
You are, however, correct that the correct fix is to have completely separate headers which define the ABI. Then the real kernel headers in
include/linuxandinclude/asmcan include them, and C libraries can also use them without contamination.
First, is there anything in include/asm that is actually needed outside the kernel itself? There shouldn't be, and if there is, it needs to be moved to a header under include/linux that is included in the relevant include/asm file.
Assuming that has been done, the first step in making this change would be to create the directory all this is to go into. Any suggestions ???
My suggestion would be to move the current headers to a new directory and put the public headers in their place. This would produce the following include directory layout:
* include/linux/ The "sanitised" header files that are
safe for use by external software.
* include/kernel/ Versions of those header files that are
used by the kernel, each including the relevant sanitised header files.
* include/asm/ Header files used only by the kernel,
all including the relevant sanitised header files.
The basic rule would be that external software can make free use of headers under include/linux but should never make any use whatsoever of headers under include/kernel or include/asm for any reason.
If anything external to the kernel needs to refer to something only defined under include/kernel/ then they should report that here for consideration as something needing a sanitised version and we could thus ensure that any oversights were dealt with.
Quoted 5 lines
This requires that someone sit down and cut'n'paste large amounts of structures and definitions from
include/linux/*.hinto the new header files. I've been tempted to do that on occasion but what's held me up has been the fact that there isn't yet a consensus on how it should be laid out.
Personally, I'm of the opinion that this will only be resolved when somebody sits down and just does it as they see fit.
Quoted 9 lines
For compatibility with older libc, one approach would be to add a new directory to the include path which matches the existing layout (linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next from the actual kernel headers to pull in those files.
Alternatively, we could go further and take the opportunity to rearrange stuff further; I'm not sure what we really gain from that though other than extra pain.
I personally can't see the point of that.
Quoted 7 lines
If Linus would approve a strategy for rearranging the headers such that people can work on it without suspecting that they're just wasting their time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait accompli -- if it isn't quite right, you end up having to do the whole thing from scratch, afaict.
The only way I can see this ever being done is one header at a time. I'm willing to have a go at doing this along the lines of the above scheme, and the first patch would be one to create the directory with a set of empty files therein.
Best wishes from Riley.
---
* Nothing as pretty as a smile, nothing as ugly as a frown.
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.481 / Virus Database: 277 - Release Date: 13-May-2003
Quoted 4 lines · Riley Williams
First, is there anything in
include/asmthat is actually needed outside the kernel itself? There shouldn't be, and if there is, it needs to be moved to a header underinclude/linuxthat is included in the relevantinclude/asmfile.
It's handy for stuff like getting high res timestamps without having to ifdef the case of building on different architechtures. Also, there are files under include/linux which end up including asm stuff in turn.
Quoted 4 lines
The basic rule would be that external software can make free use of headers under
include/linuxbut should never make any use whatsoever of headers underinclude/kernelorinclude/asmfor any reason.
What if the include/linux files themselves make use of the asm files?
Chris
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 9 lines · Chris Friesen
Quoted 4 lines · Riley Williams
First, is there anything in
include/asmthat is actually needed outside the kernel itself? There shouldn't be, and if there is, it needs to be moved to a header underinclude/linuxthat is included in the relevantinclude/asmfile.It's handy for stuff like getting high res timestamps without having to ifdef the case of building on different architechtures. Also, there are files under
include/linuxwhich end up including asm stuff in turn.
Quoted 8 lines
Quoted 4 lines
The basic rule would be that external software can make free use of headers under
include/linuxbut should never make any use whatsoever of headers underinclude/kernelorinclude/asmfor any reason.What if the
include/linuxfiles themselves make use of the asm files?
No, not acceptable.
The thing is, trying to redefine the old namespaces is hopeless at this point. Hence the proposed new namespace <linux/abi/*.h> ... <linux/abi/arch/*.h> would be my preference for an arch-specific subnamespace.
Thus the rule is:
a) <linux/abi/*> files MUST NOT include files outside <linux/abi/*>
b) <linux/*.h> and <asm/*.h> are legacy namespaces. They should be considered to be completely different in kernel and userspace -- in effect, glibc will eventually ship with its own set of these headers.
c) <linux/abi/*> files should be clean for inclusion from either kernel or userspace.
-hpa
Quoted 12 lines · H. Peter Anvin
Quoted 2 lines
What if the
include/linuxfiles themselves make use of the asm files?No, not acceptable.
The thing is, trying to redefine the old namespaces is hopeless at this point. Hence the proposed new namespace <linux/abi/*.h> ... <linux/abi/arch/*.h> would be my preference for an arch-specific subnamespace.
Thus the rule is:
Quoted 9 lines
a) <linux/abi/*> files MUST NOT include files outside <linux/abi/*>
b) <linux/*.h> and <asm/*.h> are legacy namespaces. They should be considered to be completely different in kernel and userspace -- in effect, glibc will eventually ship with its own set of these headers.
c) <linux/abi/*> files should be clean for inclusion from either kernel or userspace.
The only issue that we might have, is that <linux/abi/*> will once again break many things. Sure, if we have to fix them once to get this fixed for good, why not.
On the other hand, why not leave it at <linux/*.h> and <asm/*.h> as the location of the ABI, and then move all kernel only related stuff to <kernel/*.h> (or whatever, just the concept which count ...) which can then include whatever it needs form the other places (linux/asm)?
Regards,
Signature · 1 line
Martin Schlemmer
Quoted 11 lines · Martin Schlemmer
The only issue that we might have, is that <linux/abi/*> will once again break many things. Sure, if we have to fix them once to get this fixed for good, why not.
On the other hand, why not leave it at <linux/*.h> and <asm/*.h> as the location of the ABI, and then move all kernel only related stuff to <
kernel/*.h> (or whatever, just the concept which count ...) which can then include whatever it needs form the other places (linux/asm)?
Because then there will be a push to be backward compatible, and the current structure is hideously poor for this purpose.
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 9 lines · Martin Schlemmer
The only issue that we might have, is that <linux/abi/*> will once again break many things. Sure, if we have to fix them once to get this fixed for good, why not.
On the other hand, why not leave it at <linux/*.h> and <asm/*.h> as the location of the ABI, and then move all kernel only related stuff to <kernel/*.h> (or whatever, just the concept which count ...) which can then include whatever it needs form the other places (linux/asm)?
Quoted 2 lines
Because then there will be a push to be backward compatible, and the current structure is hideously poor for this purpose.
In fact all but the most kernel dependent programs should still not be using these headers directly.
Given that glibc plays interesting tricks with dev_t and friends. Only an application that knows what it is doing should bypass glibcs abstraction layer.
Mostly these headers will be there for glibc to include, and rexport in sys/*
Eric
Quoted 2 lines · H. Peter Anvin
What I find truly puzzling is that obviously intelligent people like yourself still seem to think that ABIs remain fixed.
Because with a few deep system stuff exceptions they do, although they certainly extend. Rogue for 0.98.5 still runs on 2.4.21 (although you may have fun finding libc2.2.2)
Quoted 5 lines · Alan Cox
Quoted 1 line
seem to think that ABIs remain fixed.
Because with a few deep system stuff exceptions they do, although they certainly extend. Rogue for 0.98.5 still runs on 2.4.21 (although you may have fun finding libc2.2.2)
ftp://ftp.win.tue.nl/pub/linux-local/libcbin/libc-2.2.2/ ?
Quoted 5 lines · Andries Brouwer
Quoted 1 line · Alan Cox
(although you may have fun finding libc2.2.2)
ftp://ftp.win.tue.nl/pub/linux-local/libcbin/libc-2.2.2/ ?
Well, no. This URL doesn't exist. I guess you meant ftp://ftp.win.tue.nl/pub/linux-local/libc.archive/libcbin/libc-2.2.2/
Best regards,
Signature · 1 line
Lionel Elie Mamane
Quoted 8 lines · Alan Cox
Quoted 2 lines · H. Peter Anvin
What I find truly puzzling is that obviously intelligent people like yourself still seem to think that ABIs remain fixed.
Because with a few deep system stuff exceptions they do, although they certainly extend. Rogue for 0.98.5 still runs on 2.4.21 (although you may have fun finding libc2.2.2)
That's my point, though: the ABI *as a whole* does not remain fixed, even though it evolves according to rules, one of which is try to maintain backwards compatibility.
The "copy-and-modify" mantra doesn't take that into account, nevermind the GPL issues.
-hpa
How about not passing the buck off to "the vendor" and helping "this vendor" make sanitized headers.
Not everybody copies RH and changes the text files from "RH" to "my distribution name".
Gentoo is my very strongly preferred distribution and I have a lot of respect and admiration for Martin for stepping in this bull*shit pool and trying to get something accomplished.
David
Quoted 13 lines · Christoph Hellwig
Quoted 5 lines · Martin Schlemmer
Point is just that people like you keep on bitching about not using sanitized kernel headers, but do nothing about it, or until today have said nothing about 'sanitized headers'.
Why don't you just get the glibc-kernheaders package from rawhide (or the equivalent from your prefered distribution) and stop complaining?
Quoted 2 lines · David Ford
How about not passing the buck off to "the vendor" and helping "this vendor" make sanitized headers.
Wjhy? Someone already did the work so why should I help to duplicate the effort. Something doesn't have to be bad just because it's from redhat.
Quoted 2 lines
Not everybody copies RH and changes the text files from "RH" to "my distribution name".
It would certainly help gentoo..
Quoted 2 lines · David Ford
How about not passing the buck off to "the vendor" and helping "this vendor" make sanitized headers.
Nothing stops gentoo contributing - or you
Or for anyone to explain or assist in the matter.
Quoted 9 lines · Alan Cox
Quoted 3 lines · David Ford
How about not passing the buck off to "the vendor" and helping "this vendor" make sanitized headers.
Nothing stops gentoo contributing - or you
The question still remains unanswered. How do you get the sanitized headers? "Use your vendor" is NOT the answer. People need to get off their high horses and stop passing the buck to the ambiguous "vendor". Even the vendor (gentoo in this case) needs the answer on how to make sanitized headers.
We are nearly 70 versions into 2.5 and approaching 2.6 quickly. There are numerous userland programs that are now using kernel headers because 2.4 headers simply don't suffice anymore -- they don't support the hardware or interfaces of today's equipment. It used to be "use the headers that you compiled the system with [glibc]" but that doesn't even work anymore.
We don't need the bullsh*t answer "don't use kernel headers" because there are oodles of packages now that require the information that is in kernel headers for one reason or another, some fishy, some legit. In a perfect world we'd have a nice set of APIs for everybody that was always kept freshly up to date from all the maintainers. If it isn't centralized and synchronized in the kernel headers there every package out there is going to be maintaining patches against the kernel source so their module builds correctly and this leads to patch co-existence nightmares.
Someone please step up to the plate and explain how to convert kernel headers into sanitized headers for /usr/include.
Thank you and flames > /dev/null
David
Quoted 5 lines
They don't. You can run the same userspace on a wide range of kernels. I'd just leave the job of selcting your headers to the distro vendor - if they are too stupid to get their headers sanitized I'd just use a different distro.
Quoted 2 lines · David Ford
Someone please step up to the plate and explain how to convert kernel headers into sanitized headers for
/usr/include.
It seems you totally miseed the entire point.
It shouldn't be an automatic conversion. It should be a well thought subset cleaned from kernel private stuff.
I maintain such a subset for my employer and it's free for all to use (it's GPL after all).
I didn't miss the point. I don't use RH, and I'm not in a mood to switch to RH just because RH has an LK headers maintainer.
The question is how to make these headers. Who decides what should and shouldn't be available to userland? What of the myriad of tools which make modules, or use deep kernel headers? What of the packages that try hard to keep subset headers synchronized but get frustrated because update patches get dropped? What am I supposed to do when I want to use several of these packages and their updates conflict with each other?
AFAIK, you don't have a package that contains all the -current- headers for all the current versions of all these various projects applied to the kernel headers and then sanitized. I want to use my hardware that is supported by version X of the package's software but the headers only have version M supported. Wireless extensions for example.
With everybody maintaining separate headers things get messy.
The question is how to make these headers. Nobody wants to say how and when someone needs an answer, even a distro maintainer, the answer is a flippant "don't use kernel headers / use your vendor". I haven't seen otherwise and believe me, I would latch on to the answer because I'm always having to tailer headers to make things work for a variety of distributions.
David
Quoted 16 lines · Arjan van de Ven
Quoted 3 lines · David Ford
Someone please step up to the plate and explain how to convert kernel headers into sanitized headers for
/usr/include.It seems you totally miseed the entire point.
It shouldn't be an automatic conversion. It should be a well thought subset cleaned from kernel private stuff.I maintain such a subset for my employer and it's free for all to use (it's GPL after all).
Quoted 1 line · David Ford
I didn't miss the point.
The rest of your mail suggests otherwise
Quoted 2 lines
I don't use RH, and I'm not in a mood to switch to RH just because RH has an LK headers maintainer.
It;s not about using RH. At all. You obviously didn't read my mail. First you say "nobody is doing the work" to which I respond "I am, and you even don't have to use RH to benifit from it".
Quoted 2 lines
The question is how to make these headers. Who decides what should and shouldn't be available to userland?
For the header set I maintain, I did. Based on the criterium "does this describe a kernel<->userland ABI".
Quoted 3 lines
AFAIK, you don't have a package that contains all the -current- headers for all the current versions of all these various projects applied to the kernel headers and then sanitized.
You really don't get it.
If a userspace app needs something REALLY special from headers, it should bloody well come with that header.
Quoted 3 lines
I want to use my hardware that
is supported by version X of the package's software but the headers only have version M supported. Wireless extensions for example.
Ok to take your example: the wireless extension using apps should include THEIR header for the extensions THEIR released userland is for, unless they want to use the sanitized headers (which should have latest). The kernel<->userspace ABI is stable, and compatible between kernel versions.
Quoted 4 lines
The question is how to make these headers. Nobody wants to say how and when someone needs an answer, even a distro maintainer, the answer is a flippant "don't use kernel headers / use your vendor". I haven't seen otherwise
I tried several times to tell you, you just don't want to hear the answer it seems.
Quoted 7 lines · Arjan van de Ven
Quoted 2 lines
I don't use RH, and I'm not in a mood to switch to RH just because RH has an LK headers maintainer.
It;s not about using RH. At all. You obviously didn't read my mail. First you say "nobody is doing the work" to which I respond "I am, and you even don't have to use RH to benifit from it".
Much appreciated for the work already done.
Quoted 12 lines
Quoted 3 lines
AFAIK, you don't have a package that contains all the -current- headers for all the current versions of all these various projects applied to the kernel headers and then sanitized.
You really don't get it.
If a userspace app needs something REALLY special from headers, it should bloody well come with that header.Quoted 3 lines
I want to use my hardware that
is supported by version X of the package's software but the headers only have version M supported. Wireless extensions for example.
Quoted 6 lines
Ok to take your example: the wireless extension using apps should include THEIR header for the extensions THEIR released userland is for, unless they want to use the sanitized headers (which should have latest). The kernel<->userspace ABI is stable, and compatible between kernel versions.
Another question that comes up ... with the many API changes from 2.4 to 2.5 ... how many issues is there with new sys calls missing, etc (sorry, only got home now.).
Quoted 8 lines
Quoted 5 lines
The question is how to make these headers. Nobody wants to say how and when someone needs an answer, even a distro maintainer, the answer is a flippant "don't use kernel headers / use your vendor". I haven't seen otherwise
I tried several times to tell you, you just don't want to hear the answer it seems.
I think on the one hand the question is also ... how far will a developer of one distro go to help another. I cannot say that I have had much success in the past to get a response from one of the 'big guys' to help me/us (the 'small guys') =)
Might it be a good thing to start an official project for this ?
Thanks,
Signature · 1 line
Martin Schlemmer
Followup to: <[email protected]> By author: Martin Schlemmer <[email protected]> In newsgroup: linux.dev.kernel
Quoted 8 lines
I think on the one hand the question is also ... how far will a developer of one distro go to help another. I cannot say that I have had much success in the past to get a response from one of the 'big guys' to help me/us (the 'small guys') =3D)
Might it be a good thing to start an official project for this ?
Yes. It's sometimes referenced as the "ABI header" project, and is the only way out of this mess.
"Copy and cleanup" is not maintainable, so then you end up having to make changes in multiple places every time. Not a good idea. Thus, what we need is ABI headers, which can be included from both kernel and userspace. This is different from the current situation, in which userspace includes random kernel headers and hopes someone has put #ifdef __KERNEL__ in all the right places. Not sustainable.
Constructing those ABI headers is going to take a lot of effort. For example <linux/abi/types.h> shouldn't export dev_t, it should export something like __kernel_dev64_t, and the structures that uses these things should be adjusted accordingly. The user-space library might include this from <sys/types.h> and have typedef __kernel_dev64_t dev_t, but that's user-space policy.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Quoted 3 lines · Arjan van de Ven
I maintain such a subset for my employer and it's free for all to use (it's GPL after all).
Do you have other tools, except good judment to help you in the process? Some perl scripts or other goodies maybe.
Sam
Quoted 8 lines · Sam Ravnborg
Quoted 3 lines · Arjan van de Ven
I maintain such a subset for my employer and it's free for all to use (it's GPL after all).
Do you have other tools, except good judment to help you in the process? Some perl scripts or other goodies maybe.
there is unifdef floating around that I'm going to use from now on ;) (I just recently found it somewhere, google for it)