Documentation
Hi,
as we talked about recruiting in the other thread I realized that one thing that might reduce the entrance barrier a bit (apart from tooling and flow) would be proper documentation.
I know documenation is usually not a developers most favorite task, but here are my findings anyway:
I had a quick look into our guideline (Documentation/kernel-doc-nano- HOWTO.txt) and it states:
1)
Quoted 2 lines
We definitely need kernel-doc formatted documentation for functions that are exported to loadable modules using
EXPORT_SYMBOL.
This is definitely not the case at the moment :/ -- there are a lot of undocumented EXPORT_SYMBOL functions.
2)
Quoted 3 lines
We also look to provide kernel-doc formatted documentation for functions externally visible to other kernel files (not marked "static").
Even less
3)
Quoted 4 lines
We also recommend providing kernel-doc formatted documentation for private (file "static") routines, for consistency of kernel source code layout. But this is lower priority and at the discretion of the MAINTAINER of that kernel source file.
Don't even talk about that :)
Imho new code should definitely have this documentation, especially for #1 and #2 -- but maybe even for #3.
So this leads me to following questions:
- How can we easily identify missing documentation?
-- Maybe Julia can come up with some coccinelle magic?
-- Maybe even mark non-extractable documentation and convert it. -- In the document it mentions scripts/basic/doproc.c checks for missing documentation, but this file does not exist anymore :/
- how can we (automatically) support documentation?
-- Can / should we create a tool that annotates functions and data types with a template documentation (I know good editors do something similar, but is the proposed style in concordance with the kernel doc style? At least it looks quite a lot like doxygen?)
- How different is the kernel doc style to doxygen?
- We are constantly looking for tasks newcomers can take on -- writing appropriate documentation is usually not that hard on the one hand and on the other people will learn a lot about the code they are documenting. -- maybe make it easier make use of Steven Rostedts proposed Kernel Patch submitting web form :P
- new checkpatch option for checking the existance of documentation? *ducks and runs away*
- and who would be in to join me in documenting everything :P
Thanks,
Peter
Quoted 6 lines
So this leads me to following questions:
- How can we easily identify missing documentation?
-- Maybe Julia can come up with some coccinelle magic?
-- Maybe even mark non-extractable documentation and convert it. -- In the document it mentionsscripts/basic/doproc.cchecks for missing documentation, but this file does not exist anymore :/
Interesting idea. I had not thought of this. Coccinelle doesn't really process comments, but one can always use grep. So the idea would be to find function definitions that don't have anything that looks like a comment in the lines above (ie, the lines since the end of the previous function definition). One could furthermore rank the results by the number of non-local calls to the function. I guess there would be a lot of reports, and it would be most productive to start with functions that are commonly used,
julia
Em Sun, 12 Jul 2015 19:15:32 -0400 (EDT)
Julia Lawall <[email protected]> escreveu:
Quoted 7 lines
Quoted 6 lines
So this leads me to following questions:
- How can we easily identify missing documentation?
-- Maybe Julia can come up with some coccinelle magic?
-- Maybe even mark non-extractable documentation and convert it. -- In the document it mentionsscripts/basic/doproc.cchecks for missing documentation, but this file does not exist anymore :/
Quoted 8 lines
Interesting idea. I had not thought of this. Coccinelle doesn't really process comments, but one can always use grep. So the idea would be to find function definitions that don't have anything that looks like a comment in the lines above (ie, the lines since the end of the previous function definition). One could furthermore rank the results by the number of non-local calls to the function. I guess there would be a lot of reports, and it would be most productive to start with functions that are commonly used,
I guess it should first try to classify the type of the exported symbols, in order to check either if they're used only internally inside a subsystem or global wide, and report, by default, just the global wide undocumented ones, sorted by number of occurrences.
Maybe the tool could be based on LXR[1] or use some output of it.
[1] http://free-electrons.com/pub/source/lxr-0.3.1-fe1.tar.bz2
Regards,
Mauro
Quoted 3 lines · Peter Hüwe
as we talked about recruiting in the other thread I realized that one thing that might reduce the entrance barrier a bit (apart from tooling and flow) would be proper documentation.
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing vertical whitespace:
/**
* is_module_percpu_address - test whether address is from module
static percpu
* @addr: address to test
*
* Test whether @addr belongs to module static percpu area. *
* RETURNS:
* %true if @addr is from module static percpu area
*/
bool is_module_percpu_address(unsigned long addr) {
I have counter suggestion:
* remove kernel-doc official status, remove generating scripts because it doesn't work,
* trim redundant and semi-redundant kernel-doc comments like this:
/**
* module_refcount - return the refcount or -1 if unloading *
* @mod: the module we're checking
*
* Returns:
* -1 if the module is in the process of unloading
* otherwise the number of references in the kernel to the module
*/
int module_refcount(struct module *mod)
{
("return reference count or -1 if module is being unloaded" is enough here).
Alexey
Quoted 21 lines · Alexey Dobriyan
Quoted 3 lines · Peter Hüwe
as we talked about recruiting in the other thread I realized that one thing that might reduce the entrance barrier a bit (apart from tooling and flow) would be proper documentation.
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing vertical whitespace:
/**
*is_module_percpu_address- test whether address is from module
static percpu
* @addr: address to test
*
* Test whether @addr belongs to module static percpu area. *
* RETURNS:
* %true if @addr is from module static percpu area
*/
Quoted 6 lines
bool
is_module_percpu_address(unsigned long addr){I have counter suggestion:
* remove kernel-doc official status, remove generating scripts because it doesn't work,
I have no problem with that suggestion. scripts/kernel-doc is fragile and either needs lots of continuous help or can be discarded.
As I have said in the past, the main thing is that there should be decent documentation for (new) interfaces. It doesn't need to be kernel-doc.
Quoted 15 lines
* trim redundant and semi-redundant kernel-doc comments like this:
/**
*module_refcount- return the refcount or -1 if unloading *
* @mod: the module we're checking
*
* Returns:
* -1 if the module is in the process of unloading
* otherwise the number of references in the kernel to the module
*/
intmodule_refcount(struct module *mod)
{("return reference count or -1 if module is being unloaded" is enough here).
Signature · 1 line
~Randy
Quoted 4 lines · On Mon, 13 Jul 2015 12:47:23 +0300 Alexey Dobriyan <[email protected]> wrote:
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing vertical whitespace:
I agree that newbies shouldn't write documentation. But maybe what they can do is to question what a function does. And perhaps poke the maintainer (or author of said function) to write something that explains that function (only for non-static functions).
Now, I say "newbies" but I would really mean experienced developers that are new to a subsystem. We don't need silly questions. Something more on the line of one experienced kernel developer reading some code of the kernel they have no idea about, and if they can't figure out what a function does, ask the question to the author. Perhaps we can get better documentation of internal interfaces out of it.
There's been times where I look at other subsystems and wonder WTF a function is suppose to do. Perhaps I need to start poking the author of that code to explain it.
-- Steve
Quoted 3 lines · Steven Rostedt
There's been times where I look at other subsystems and wonder WTF a function is suppose to do. Perhaps I need to start poking the author of that code to explain it.
That should never happen. It's OK if you just don't know how to do something (or don't even know if it's possible), and you have to ask or resort to documentation.
But you, Steven, should *never* find yourself looking at a function and wondering WTF it does. If a function is complex enough that you need an explanation, then either:
1. It should have been made simpler, or
2. There should be an explanation *right* there in the C file.
Those should be caught in review. Too often, I think, we allow opaque code to get merged.
I'm guilty of it myself. And I've also suffered from it. Sometimes, with a few years in between, I have even inflicted this pain on *myself* :)
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 3 lines · On Mon, 13 Jul 2015 17:46:46 +0100 David Woodhouse <[email protected]> wrote:
I'm guilty of it myself. And I've also suffered from it. Sometimes, with a few years in between, I have even inflicted this pain on *myself* :)
Me too. I have to admit, that I've done the same with my own functions. But I've become much better lately at documenting.
A lot of these functions are from years past. We should go back and try to document them. Perhaps we may even find and fix a few bugs in doing so.
I forgot who said it, but I remember reading a quote that stated "When the code does not match the comments, they are probably both wrong".
-- Steve
Am Montag, 13. Juli 2015, 19:35:19 schrieb Steven Rostedt:
Quoted 3 lines
A lot of these functions are from years past. We should go back and try to document them. Perhaps we may even find and fix a few bugs in doing so.
+100!
Quoted 3 lines
I forgot who said it, but I remember reading a quote that stated "When the code does not match the comments, they are probably both wrong".
+1000! :)
Of course that's a huge problem, even more with our fast paced development.
I think we should definitely spend more time to fix stuff/documentation we stumble upon and spent hours figuring out anyway.
Peter
Am Montag, 13. Juli 2015, 18:46:46 schrieb David Woodhouse:
Quoted 4 lines
1. It should have been made simpler, or
2. There should be an explanation *right* there in the C file.Those should be caught in review.
But also we have tons of undocumented obscure legacy code which unfortunately won't get new review cycles.
Peter
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 13 Jul 2015 12:47:23 +0300 Alexey Dobriyan <[email protected]> wrote:
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing vertical whitespace:
I agree that newbies shouldn't write documentation.
I semi-disagree. Assuming you meant newbies here as you defined below.
Not that an exception should make the rule, but please take a look at the Crypto API documentation (with userspace example code!) written by Stephan Mueller:
git log --author="[email protected]"
His first patch was in May 2014.
541af946fe13 crypto: drbg - SP800-90A Deterministic Random Bit Generator
By Nov/Dec 2014 he had documented the entire crypto API.
52744af3af97 crypto: doc - document uncovered member variables 47ca5be9eb06 crypto: doc - HASH API documentation 16e61030aecb crypto: doc - CIPHER API documentation 58284f0d6c4a crypto: doc - BLKCIPHER API documentation fced7b02623e crypto: doc - AEAD API documentation f13ec330a787 crypto: doc - ABLKCIPHER API documentation 0d7f488f0305 crypto: doc - cipher data structures 968ab2910780 crypto: doc - SHASH API documentation 90240ffb1277 crypto: doc - AHASH API documentation 5d8c723f61f2 crypto: doc - hash data structures aa1b6fbcbeac crypto: doc - RNG API documentation e63b673f601d crypto: doc - userspace interface spec e9a44230dbca crypto: doc - compile crypto API spec 7d12993ed890 crypto: doc - crypto API high level spec
Often the best person to write the docs for a newcomer to understand is someone who was just recently a newcomer.
Quoted 4 lines
But maybe what they
can do is to question what a function does. And perhaps poke the maintainer (or author of said function) to write something that explains that function (only for non-static functions).
Submitting patches is often the *best* way to poke a maintainer. ;-)
I know I'm not the only one who would find it a lot easier to tweak an 80% correct documentation patch as opposed to explaining the function call, how it fits in with the others, how it historically evolved that way, etc.
Quoted 6 lines
Now, I say "newbies" but I would really mean experienced developers that are new to a subsystem. We don't need silly questions. Something more on the line of one experienced kernel developer reading some code of the kernel they have no idea about, and if they can't figure out what a function does, ask the question to the author. Perhaps we can get better documentation of internal interfaces out of it.
thx,
Jason.
Quoted 5 lines · On Mon, 13 Jul 2015 17:42:44 +0000 Jason Cooper <[email protected]> wrote:
Quoted 1 line · Steven Rostedt
I agree that newbies shouldn't write documentation.
I semi-disagree. Assuming you meant newbies here as you defined below.
This really shows what the issue with new recruits is all about. Every maintainer is different. The Linux kernel is the largest and fastest open source project in the world and its amazing how much that everyone does agree on. But every maintainer has a slightly different perspective of how to do something, and this can be really aggravating to a new comer. As it's been said lots of times, maintaining the Linux kernel is much like herding cats. And this burden also weighs down on new comers, as they try to figure out which type of cat they are dealing with. And all cats just happen to be extremely annoying to deal with ;-)
The best we can do is all to try to be patient with those that contact us.
-- Steve
Quoted 20 lines · Steven Rostedt
Quoted 5 lines · On Mon, 13 Jul 2015 17:42:44 +0000 Jason Cooper <[email protected]> wrote:
Quoted 1 line · Steven Rostedt
I agree that newbies shouldn't write documentation.
I semi-disagree. Assuming you meant newbies here as you defined below.
This really shows what the issue with new recruits is all about. Every maintainer is different. The Linux kernel is the largest and fastest open source project in the world and its amazing how much that everyone does agree on. But every maintainer has a slightly different perspective of how to do something, and this can be really aggravating to a new comer. As it's been said lots of times, maintaining the Linux kernel is much like herding cats. And this burden also weighs down on new comers, as they try to figure out which type of cat they are dealing with. And all cats just happen to be extremely annoying to deal with ;-)
Quoted 3 lines
The best we can do is all to try to be patient with those that contact us.
I think it's more often that we just ignore them rather than being impatient...
Quoted 7 lines
Quoted 1 line
I agree that newbies shouldn't write documentation.
I semi-disagree. Assuming you meant newbies here as you defined below.
Not that an exception should make the rule, but please take a look at the Crypto API documentation (with userspace example code!) written by Stephan Mueller:
Thanks Jason for this example -- of course that's an (exceptional) exception to the rule, I really like it anyway.
Quoted 2 lines
Often the best person to write the docs for a newcomer to understand is someone who was just recently a newcomer.
:) definitely.
I mentor quite a lot of new colleagues at my company, and you cannot image how many 'blind spots' new colleagues can find -- even in 'perfect' documentation/tutorials.
Thanks
Peter
Quoted 38 lines · On Monday 13 July 2015 17:42:44 Jason Cooper wrote:
Quoted 7 lines · Steven Rostedt
Quoted 5 lines · On Mon, 13 Jul 2015 12:47:23 +0300 Alexey Dobriyan wrote:
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing
vertical whitespace:
I agree that newbies shouldn't write documentation.
I semi-disagree. Assuming you meant newbies here as you defined below.
Not that an exception should make the rule, but please take a look at the Crypto API documentation (with userspace example code!) written by Stephan Mueller:
git log --author="[email protected]"His first patch was in May 2014.
541af946fe13crypto: drbg - SP800-90A Deterministic Random Bit GeneratorBy Nov/Dec 2014 he had documented the entire crypto API.
52744af3af97crypto: doc - document uncovered member variables47ca5be9eb06crypto: doc - HASH API documentation16e61030aecbcrypto: doc - CIPHER API documentation58284f0d6c4acrypto: doc - BLKCIPHER API documentationfced7b02623ecrypto: doc - AEAD API documentationf13ec330a787crypto: doc - ABLKCIPHER API documentation0d7f488f0305crypto: doc - cipher data structures968ab2910780crypto: doc - SHASH API documentation90240ffb1277crypto: doc - AHASH API documentation5d8c723f61f2crypto: doc - hash data structuresaa1b6fbcbeaccrypto: doc - RNG API documentatione63b673f601dcrypto: doc - userspace interface spece9a44230dbcacrypto: doc - compile crypto API spec7d12993ed890crypto: doc - crypto API high level spec
Quoted 2 lines
Often the best person to write the docs for a newcomer to understand is someone who was just recently a newcomer.
I agree. Not that I want to show off, but there's at least another example.
9cad9c95d7e8 Documentation: DocBook DRM framework documentation
Quoted 18 lines
Quoted 4 lines
But maybe what they
can do is to question what a function does. And perhaps poke the maintainer (or author of said function) to write something that explains that function (only for non-static functions).Submitting patches is often the *best* way to poke a maintainer. ;-)
I know I'm not the only one who would find it a lot easier to tweak an 80% correct documentation patch as opposed to explaining the function call, how it fits in with the others, how it historically evolved that way, etc.
Quoted 6 lines
Now, I say "newbies" but I would really mean experienced developers that are new to a subsystem. We don't need silly questions. Something more on the line of one experienced kernel developer reading some code of the kernel they have no idea about, and if they can't figure out what a function does, ask the question to the author. Perhaps we can get better documentation of internal interfaces out of it.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 3 lines · On Mon, 13 Jul 2015 12:47:23 +0300 Alexey Dobriyan <[email protected]> wrote:
I have counter suggestion:
* remove kernel-doc official status, remove generating scripts because it doesn't work,
kerneldoc does work, after a fashion. It works well enough that people use it, and I hear about it quickly when somebody's comment change breaks the docs build. I don't think we can remove it in the absence of something better.
Creating "something better" is actually on my list of things to do. The only problem is that I've had to buy a second 4TB drive to hold that list, so I don't know when I'll have time to think about it.
jon
Am Montag, 13. Juli 2015, 19:45:09 schrieb Jonathan Corbet:
Quoted 6 lines
On Mon, 13 Jul 2015 12:47:23 +0300
Quoted 3 lines · Alexey Dobriyan
I have counter suggestion:
* remove kernel-doc official status, remove generating scripts
Quoted 6 lines
Quoted 1 line
because it doesn't work,
kerneldoc does work, after a fashion. It works well enough that people use it, and I hear about it quickly when somebody's comment change breaks the docs build. I don't think we can remove it in the absence of something better.
Jonathan, do you keep an up-to-date copy of the generated files somewhere? (I cannot get it to work out of the box on my machine)
Quoted 3 lines
Creating "something better" is actually on my list of things to do. The only problem is that I've had to buy a second 4TB drive to hold that list, so I don't know when I'll have time to think about it.
Where do you think are the major issues from your point of view?
Would doxygen be "something better"? The kernel-doc style looks already quite similar.
Maybe the kernel should not reinvent everything :)
Thanks,
Peter
Quoted 6 lines · On Mon, 13 Jul 2015 21:46:46 +0200 Peter Hüwe <[email protected]> wrote:
Quoted 4 lines
kerneldoc does work, after a fashion. It works well enough that people use it, and I hear about it quickly when somebody's comment change breaks the docs build. I don't think we can remove it in the absence of something better.
Jonathan, do you keep an up-to-date copy of the generated files somewhere? (I cannot get it to work out of the box on my machine)
I don't, sorry. That would probably be a good thing for me to do for a number of reasons.
Quoted 4 lines
Quoted 3 lines
Creating "something better" is actually on my list of things to do. The only problem is that I've had to buy a second 4TB drive to hold that list, so I don't know when I'll have time to think about it.
Where do you think are the major issues from your point of view?
Well, as mentioned before, kerneldoc is fragile. A little while back somebody changed a struct to a union without updating the comments to match; that was enough to break the entire document build process. That kind of thing happens a lot; I don't think it should have to be that way.
Beyond that, DocBook is a major pain. I get a fair number of patches along the lines of "that </para> should really have been ahead of that </whatever>". It's verbose, intimidating to newcomers, and causes more worn-out shift keys than anything else. We don't need something with the complexity of DocBook; something closer to Markdown or ReST would do us just fine and make the documentation much more accessible.
But making any such change is a big job, and convincing the community to go with a change in tooling is probably a bigger job. So I'm not rushing into it.
Quoted 3 lines
Would doxygen be "something better"? The kernel-doc style looks already quite similar.
Maybe the kernel should not reinvent everything :)
Doxygen may be worth a look. I'm personally fond of Sphinx, though I still haven't done a big project in it.
But this is all just dreaming until somebody has the time to do a major docs overhaul and sell it to everybody else. Until then, I'm focused on slowly improving what we have now...
jon
(CC'ing Daniel Vetter)
Quoted 37 lines · On Monday 13 July 2015 20:36:47 Jonathan Corbet wrote:
Quoted 7 lines · On Mon, 13 Jul 2015 21:46:46 +0200 Peter Hüwe wrote:
Quoted 4 lines
kerneldoc does work, after a fashion. It works well enough that people use it, and I hear about it quickly when somebody's comment change breaks the docs build. I don't think we can remove it in the absence of something better.
Jonathan, do you keep an up-to-date copy of the generated files somewhere? (I cannot get it to work out of the box on my machine)
I don't, sorry. That would probably be a good thing for me to do for a number of reasons.
Quoted 5 lines
Quoted 3 lines
Creating "something better" is actually on my list of things to do. The only problem is that I've had to buy a second 4TB drive to hold that list, so I don't know when I'll have time to think about it.
Where do you think are the major issues from your point of view?
Well, as mentioned before, kerneldoc is fragile. A little while back somebody changed a struct to a union without updating the comments to match; that was enough to break the entire document build process. That kind of thing happens a lot; I don't think it should have to be that way.
Beyond that, DocBook is a major pain. I get a fair number of patches along the lines of "that </para> should really have been ahead of that </whatever>". It's verbose, intimidating to newcomers, and causes more worn-out shift keys than anything else. We don't need something with the complexity of DocBook; something closer to Markdown or ReST would do us just fine and make the documentation much more accessible.
But making any such change is a big job, and convincing the community to go with a change in tooling is probably a bigger job. So I'm not rushing into it.
Quoted 2 lines
Would doxygen be "something better"? The kernel-doc style looks already quite similar. Maybe the kernel should not reinvent everything :)
Quoted 6 lines
Doxygen may be worth a look. I'm personally fond of Sphinx, though I still haven't done a big project in it.
But this is all just dreaming until somebody has the time to do a major docs overhaul and sell it to everybody else. Until then, I'm focused on slowly improving what we have now...
Daniel, I think I recall you mentioning some efforts going in that direction. Any update ?
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 43 lines · Laurent Pinchart
(CC'ing Daniel Vetter)
Quoted 40 lines · On Monday 13 July 2015 20:36:47 Jonathan Corbet wrote:
Quoted 7 lines · On Mon, 13 Jul 2015 21:46:46 +0200 Peter Hüwe wrote:
Quoted 4 lines
kerneldoc does work, after a fashion. It works well enough that people use it, and I hear about it quickly when somebody's comment change breaks the docs build. I don't think we can remove it in the absence of something better.
Jonathan, do you keep an up-to-date copy of the generated files somewhere? (I cannot get it to work out of the box on my machine)
I don't, sorry. That would probably be a good thing for me to do for a number of reasons.
Quoted 5 lines
Quoted 3 lines
Creating "something better" is actually on my list of things to do. The only problem is that I've had to buy a second 4TB drive to hold that list, so I don't know when I'll have time to think about it.
Where do you think are the major issues from your point of view?
Well, as mentioned before, kerneldoc is fragile. A little while back somebody changed a struct to a union without updating the comments to match; that was enough to break the entire document build process. That kind of thing happens a lot; I don't think it should have to be that way.
Beyond that, DocBook is a major pain. I get a fair number of patches along the lines of "that </para> should really have been ahead of that </whatever>". It's verbose, intimidating to newcomers, and causes more worn-out shift keys than anything else. We don't need something with the complexity of DocBook; something closer to Markdown or ReST would do us just fine and make the documentation much more accessible.
But making any such change is a big job, and convincing the community to go with a change in tooling is probably a bigger job. So I'm not rushing into it.
Quoted 2 lines
Would doxygen be "something better"? The kernel-doc style looks already quite similar. Maybe the kernel should not reinvent everything :)
Doxygen may be worth a look. I'm personally fond of Sphinx, though I still haven't done a big project in it.
Quoted 6 lines
Quoted 3 lines
But this is all just dreaming until somebody has the time to do a major docs overhaul and sell it to everybody else. Until then, I'm focused on slowly improving what we have now...
Daniel, I think I recall you mentioning some efforts going in that direction. Any update ?
We're contracting collabora (Daniel is working on it, cc'ed) to pimp the kerneldoc toolchain for us:
- hyperlink functions/structures in the generated docbook
- add markdown support for kerneldoc paragraphs so that you can do lists, basic formatting and tables
That's the basics I really need, whith that we can probably move all the text sections from the drm docbook into the code. On top of that there's a few wishlist items:
- allow to split up per-member sections for long structs (especially useful for vtable structs where you want some longer text per each function)
- asciiart or something (box model of drm planes/crtcs isn't the simplest thing)
We do use other stuff in userspace which is orders of magnitude better than kerneldoc, but kerneldoc+docbook template has intertia. At least in drm-land, and that's why I decided it's better to stick with it.
My longer-term goal is to fully document drm and i915 internals using this, to help us bring new engineers up to speed faster. We have a massive problem with training in that area in drm/i915. I'm also trying to get a dedicated tech writer to keep all the drm docs written by various people consistent. Plan is to do that once the kerneldoc tooling is in more useful shape.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Em Mon, 13 Jul 2015 20:36:47 -0600
Jonathan Corbet <[email protected]> escreveu:
Quoted 3 lines
Quoted 3 lines
Would doxygen be "something better"? The kernel-doc style looks already quite similar.
Maybe the kernel should not reinvent everything :)
My experience is that, from time to time, people send us drivers written by the manufacturer with Doxygen-compatible tags. Converting them to kernel-doc is a huge task, not worthing the effort. I tried in the past to request developers to "fix" the comments. The net result is either to not have a final version of the driver submitted or to receive a comments-stripped version of it, with is actually worse, as we loose lots of otherwise good documentation. So, nowadays, I'm opting to keep whatever comments are there at the code, provided that it is not using C99 comments.
So, I'm a huge fan of extending the kernel-doc to accept all markup tags that Doxygen supports.
Quoted 2 lines
Doxygen may be worth a look. I'm personally fond of Sphinx, though I still haven't done a big project in it.
I never used Sphinx. Doxygen seems better on my eyes, because people already sends us media drivers using Doxygen tags style. Also, as mentioned, kernel-doc style is a subset. Not to mention that, in the specific case of media,we're using Doxygen to document some libraries packaged inside v4l-utils (with is the userspace counterpart of the Kernel subsystem). So, at least for me, Doxygen would be easier to use.
Regards,
Mauro
Quoted 9 lines · On Tue, 14 Jul 2015 09:43:42 -0300 Mauro Carvalho Chehab…
Quoted 2 lines
Doxygen may be worth a look. I'm personally fond of Sphinx, though I still haven't done a big project in it.
I never used Sphinx. Doxygen seems better on my eyes, because people already sends us media drivers using Doxygen tags style. Also, as mentioned, kernel-doc style is a subset. Not to mention that, in the specific case of media,we're using Doxygen to document some libraries packaged inside v4l-utils (with is the userspace counterpart of the Kernel subsystem). So, at least for me, Doxygen would be easier to use.
Just for clarity: any new documentation solution that required reformatting all existing doc comments is clearly not going anywhere; even I am not dumb enough to try to push something like that through.
In any case, this is all in the "would be nice to look at seriously" stage over here. I have no real idea if something like Sphinx would be a useful and practical improvement or not; I'm certainly not agitating for a change in that direction now.
jon
Quoted 21 lines · On Tuesday 14 July 2015 09:43:42 Mauro Carvalho Chehab wrote:
Em Mon, 13 Jul 2015 20:36:47 -0600 Jonathan Corbet escreveu:
Quoted 3 lines
Quoted 3 lines
Would doxygen be "something better"? The kernel-doc style looks already quite similar.
Maybe the kernel should not reinvent everything :)My experience is that, from time to time, people send us drivers written by the manufacturer with Doxygen-compatible tags. Converting them to kernel-doc is a huge task, not worthing the effort. I tried in the past to request developers to "fix" the comments. The net result is either to not have a final version of the driver submitted or to receive a comments-stripped version of it, with is actually worse, as we loose lots of otherwise good documentation. So, nowadays, I'm opting to keep whatever comments are there at the code, provided that it is not using C99 comments.
So, I'm a huge fan of extending the kernel-doc to accept all markup tags that Doxygen supports.
Quoted 2 lines
Doxygen may be worth a look. I'm personally fond of Sphinx, though I still haven't done a big project in it.
Quoted 6 lines
I never used Sphinx. Doxygen seems better on my eyes, because people already sends us media drivers using Doxygen tags style. Also, as mentioned, kernel-doc style is a subset. Not to mention that, in the specific case of media,we're using Doxygen to document some libraries packaged inside v4l-utils (with is the userspace counterpart of the Kernel subsystem). So, at least for me, Doxygen would be easier to use.
We certainly need to accept documentation formats that are easy to use and write, but let's not forget that the biggest issue (in my opinion) is that we often don't get documentation at all. Cumbersome formats and tools might explain part of it, but that's not the whole story. How can we ensure that documentation gets written, merged and kept up to date ? What kind of incentives can we give ? The stick is easy to wave by not merging code changes that are not documented, but that can easily backfire.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 15 lines · Peter Hüwe
Am Montag, 13. Juli 2015, 19:45:09 schrieb Jonathan Corbet:
Quoted 6 lines
On Mon, 13 Jul 2015 12:47:23 +0300
Quoted 3 lines · Alexey Dobriyan
I have counter suggestion:
* remove kernel-doc official status, remove generating scriptsQuoted 8 lines
Quoted 1 line
because it doesn't work,
kerneldoc does work, after a fashion. It works well enough that people
use it, and I hear about it quickly when somebody's comment change breaks
the docs build. I don't think we can remove it in the absence of something better.
Quoted 4 lines
Jonathan, do you keep an up-to-date copy of the generated files somewhere?
(I cannot get it to work out of the box on my machine)
kernel.org does, no?
https://www.kernel.org/doc/htmldocs/
johannes
Am Montag, 13. Juli 2015, 11:47:23 schrieb Alexey Dobriyan:
Quoted 9 lines · Peter Hüwe
Quoted 3 lines
as we talked about recruiting in the other thread I realized that one thing that might reduce the entrance barrier a bit (apart from tooling and flow) would be proper documentation.
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing vertical whitespace:
While I agree with your 'stupid documentation is stupid' argument, I tend to somewhat disagree about that newbies should not write documentation.
By newbies I do not mean someone new to programming - that of course makes no sense - but why shouldn't a experienced c-programmer who is currently figuring out how the kernel works not document stuff along the way?
I think that's much better than fixing whitespaces and long lines.
Peter
Quoted 10 lines · On Monday 13 July 2015 21:20:32 Peter Hüwe wrote:
Am Montag, 13. Juli 2015, 11:47:23 schrieb Alexey Dobriyan:
Quoted 8 lines · Peter Hüwe
Quoted 3 lines
as we talked about recruiting in the other thread I realized that one thing that might reduce the entrance barrier a bit (apart from tooling and flow) would be proper documentation.
Newbies should not write documentation because they, by definition, have little knowledge of the code. It will lead to many, many useless and redundant comments like below whose only purpose is stealing
Quoted 7 lines
Quoted 1 line
vertical whitespace:
While I agree with your 'stupid documentation is stupid' argument, I tend to somewhat disagree about that newbies should not write documentation.
By newbies I do not mean someone new to programming - that of course makes no sense - but why shouldn't a experienced c-programmer who is currently figuring out how the kernel works not document stuff along the way?
I fully agree, as I've gone through that process when I wrote my first DRM/KMS driver, and found it very valuable.
Writing documentation is a very good way to try and understand a subsystem for newcomers. It's time consuming though, which goes in the way of the tight schedules everybody seems to have. Furthermore employers are unfortunately not usually keen to fund documentation, especially when the schedule is tight.
On the other hand, I realized that writing down my findings while I was exploring the DRM subsystem was a great way to learn about it. It then took a bit of time to format my notes into proper documentation, but less than writing everything from scratch in one go. Taking notes also help structuring the documentation in a more natural way.
The picture wasn't all bright though. The documentation patch I submitted was of course not 100% complete, but the problem came later when the subsystem got rearchitectured without updating the documentation. Code changes were merged with kerneldoc comments to document functions, but the global DocBook architecture documentation was left untouched (except for integration of kerneldoc as API reference). The end result is that the DRM/KMS subsystem went from having a reasonable architecture documentation to being undocumented. Needless to say my motivation as a documentation writer for DRM/KMS took a big blow.
This being said, not everything is dark either. The V4L2 subsystem, for instance, does a pretty good job at keeping its documentation up-to-date. The maintainers and core developers always nack code changes that do not come with related documentation updates. This proved to be effective, but requires the rule to be strictly enforced. I wonder if this could scale to other subsystems, and what kind of other incentives we could create.
Quoted 1 line
I think that's much better than fixing whitespaces and long lines.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 9 lines · Peter Hüwe
Hi,
as we talked about recruiting in the other thread I realized that one thing that might reduce the entrance barrier a bit (apart from tooling and flow) would be proper documentation.
I know documenation is usually not a developers most favorite task, but here are my findings anyway:
Hi, all
Comparing to the documents of functions, some other kinds of documents might be more important but even seldom.
First, documents of why/what/how.
- Why the functionality/subsystem/structure/function/marco is needed?
- what is it
- How to implement it.
These documents are less and sometimes they are hidden in changelog. why/what/how is suggested in changelog, so I often use git-blame to fetch them, but it is inconvenient for normal-reviewers when it is stored outside of kernel_src/Documentation/.
Secondly, documents of architecture/design.
- architecture overview
- architecture for control/data flow
These documents are less and sometimes they don't have proper picture inside them. Kernel developers like ascii pictures, but ascii pictures can't provide enough information with decent density. and it is hard to draw, to maintain... etc.
I once borrowed the two pictures in the page7/page9 of this:
http://www.linux-kongress.org/2010/slides/KVM-Architecture-LK2010.pdf
and gave 3 hours training to someones. It helped me showed/remembered the architecture, the interfaces between the components and the major structures inside every components. It would be useful if we also have sufficient document of this kind in the kernel.
The documents of functions are also the documents of interfaces between the components, they are important, so I also support for Peter Hüwe.
And as noted before, changlog is also document for me, and IDEs(vim) have the functionality to find the definition/usage, I also hope they have the functionality to "find the changelog of this line".
Thanks,
Lai
Quoted 70 lines
I had a quick look into our guideline (
Documentation/kernel-doc-nano-HOWTO.txt) and it states:1)
Quoted 2 lines
We definitely need kernel-doc formatted documentation for functions that are exported to loadable modules using
EXPORT_SYMBOL.This is definitely not the case at the moment :/ -- there are a lot of undocumented
EXPORT_SYMBOLfunctions.2)
Quoted 3 lines
We also look to provide kernel-doc formatted documentation for functions externally visible to other kernel files (not marked "static").
Even less
3)
Quoted 4 lines
We also recommend providing kernel-doc formatted documentation for private (file "static") routines, for consistency of kernel source code layout. But this is lower priority and at the discretion of the MAINTAINER of that kernel source file.
Don't even talk about that :)
Imho new code should definitely have this documentation, especially for #1 and #2 -- but maybe even for #3.
So this leads me to following questions:
- How can we easily identify missing documentation?
-- Maybe Julia can come up with some coccinelle magic?
-- Maybe even mark non-extractable documentation and convert it. -- In the document it mentionsscripts/basic/doproc.cchecks for missing documentation, but this file does not exist anymore :/- how can we (automatically) support documentation?
-- Can / should we create a tool that annotates functions and data types with a template documentation (I know good editors do something similar, but is the proposed style in concordance with the kernel doc style? At least it looks quite a lot like doxygen?)- How different is the kernel doc style to doxygen?
- We are constantly looking for tasks newcomers can take on -- writing appropriate documentation is usually not that hard on the one hand and on the other people will learn a lot about the code they are documenting. -- maybe make it easier make use of Steven Rostedts proposed Kernel Patch submitting web form :P
- new checkpatch option for checking the existance of documentation? *ducks and runs away*
- and who would be in to join me in documenting everything :P
Thanks,
Peter
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Quoted 3 lines · On Mon, 13 Jul 2015 00:38:01 +0200 Peter Hüwe <[email protected]> wrote:
- We are constantly looking for tasks newcomers can take on -- writing appropriate documentation is usually not that hard on the one hand and on the other people will learn a lot about the code they are documenting.
If you know people who can write coherently about kernel stuff — and actually want to do so — please send them my way. I can't find such people, and I'm willing to pay for the work. I don't hold out much hope for getting newcomers to fill that gap.
jon