Skip to contentSkip to conversation

Documentation

Ksummit-discuss Jonathan Corbet 16 participants 32 messages Aug 1, 2015 – Aug 5, 2015

lore
Share & tools
lore
#0 lore raw

As your relatively new documentation subsystem maintainer, I think there might be reason to talk about docs for a bit at the summit.

Currently, we have two major branches to our docs. One, the bulk of Documentation/, is a haphazard collection of text files of varying quality and applicability to current kernels. The other is a set of more organized DocBook documents.

Despite their disorganized nature, the plain-text docs get the bulk of the attention. Anybody can manage to work with a text file. The DocBook stuff, instead, is kind of a pain. Working in DocBook itself is less than fun, few people dare to delve into the code that builds the docs, and the whole thing is rather fragile. Changing a struct to a union in the wireless subsystem a few months back broke the docs build, for example; few people test for such things and even fewer seem to know how to fix them.

We have some interesting proposals for improvements on the DocBook side, including integrating Markdown support
(http://thread.gmane.org/gmane.linux.documentation/32140) and allowing embedded DocBook comments within structs
(https://lwn.net/Articles/653194/). I think they both have merit, but I hesitate to take them for a couple of reasons.

One is that changes in how docs are done would be imposed on everybody who is interested in keeping up the docs. They also can, in a sense, look like coding-style changes, which is an area where I fear to tread on my own. The other is that I wonder if we want to consider a bit of a more sweeping overhaul.

I am not convinced that DocBook is really the right tool for the job here. It is complex, finicky, and it's a real pain to have to replace an entire laptop because you've worn out your shift key. It is, I believe, an impediment to improvements to our docs in general. If, instead, we used something more plain-text-like, I think life might get a lot easier; it could also enable the integration of all of our docs into something a bit more cohesive.

Markdown in one of its many forms might be a good alternative here. I'm also somewhat attracted by Sphinx, which is designed for documenting code already and could probably be made to work well with our existing kerneldoc comments without a whole lot of trouble. The Sphinx idea, though, is hobbled by the inconvenient fact that I've not had the time to develop it far enough to even have a vague idea of whether it would make real-world sense.

There could, I think, be value in talking about where we would like our docs subsystem to go. What kind of changes would we like to make to render our docs more accessible, more current, and easier to contribute to? Being on the hook for a KS section might also motivate me to actually develop my ideas a bit...

jon

#1 lore raw
Quoted 2 lines · Jonathan Corbet

There could, I think, be value in talking about where we would like our docs subsystem to go.

Perhaps more important would be focusing on ways of _improving_ our already existing Documentation/*. Some areas really stink, a combination of stale and incorrect information. My brain is too melted to give exact examples, just one of those things you run into.

In general core kernel docs is pretty well maintained. This, I believe, is a result of maintainers pushing for quality docs with patches. I don't know about others, ie drivers, but perhaps examples could be taken from -tip, or akpm.

Quoted 2 lines

What kind of changes would we like to make to render
our docs more accessible, more current, and easier to contribute to?

I would say that if anyone has anything meaningful to say, he or she, will find the way. As you mentioned, nicely formatted plain text files work well.

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

Thanks,
Davidlohr

#2 lore raw
Quoted 21 lines · Davidlohr Bueso
Quoted 2 lines · Jonathan Corbet

There could, I think, be value in talking about where we would like our docs subsystem to go.

Perhaps more important would be focusing on ways of _improving_ our already existing Documentation/*. Some areas really stink, a combination of stale and incorrect information. My brain is too melted to give exact examples, just one of those things you run into.

In general core kernel docs is pretty well maintained. This, I believe, is a result of maintainers pushing for quality docs with patches. I don't know about others, ie drivers, but perhaps examples could be taken from -tip, or akpm.

Quoted 2 lines

What kind of changes would we like to make to render
our docs more accessible, more current, and easier to contribute to?

I would say that if anyone has anything meaningful to say, he or she, will find the way. As you mentioned, nicely formatted plain text files work well.

Quoted 6 lines

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

"thought" I guess?

I have to say I agree here.

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

Thanks,
Rafael

#3 lore raw
Quoted 2 lines · Rafael J. Wysocki

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

FYI, we can easily turn on the "render markdown" plugin on git.kernel.org if there is actually a use for that.

Best,

Signature · 3 lines
Konstantin Ryabitsev
Linux Foundation Collab Projects
Montréal, Québec
#4 lore raw
In reply to Rafael J. Wysocki
Quoted 10 lines · On Mon, 03 Aug 2015 15:35:36 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 6 lines

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

"thought" I guess?

I have to say I agree here.

Surely nobody thinks I was saying that the documentation-writing process can be automated! :) But we go to some lengths now to document our APIs in the code; I don't think we would want to break that.

Quoted 2 lines

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

jon

#5 lore raw
Quoted 18 lines · Jonathan Corbet
Quoted 10 lines · On Mon, 03 Aug 2015 15:35:36 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 6 lines

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

"thought" I guess?

I have to say I agree here.

Surely nobody thinks I was saying that the documentation-writing process can be automated! :) But we go to some lengths now to document our APIs in the code; I don't think we would want to break that.

Quoted 6 lines
Quoted 2 lines

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

My main issue with the DocBook is that it is not generated by default and it is actually takes a while to generate them; otherwise I woudl love to be able to catch at least issues breaking existing docs as early as possible.

Thanks.

Signature · 1 line
Dmitry
#7 lore raw
In reply to Jonathan Corbet
Quoted 10 lines · Jonathan Corbet
Quoted 7 lines · On Mon, 03 Aug 2015 15:35:36 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 6 lines

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

Quoted 7 lines
Quoted 3 lines

"thought" I guess?

I have to say I agree here.

Surely nobody thinks I was saying that the documentation-writing process can be automated! :) But we go to some lengths now to document our APIs in the code; I don't think we would want to break that.

No, I wouldn't at least.

Quoted 6 lines
Quoted 2 lines

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

I must admit I'm not really familiar with any of these, but at least Markdown looks promising from a quick glance.

Thanks,
Rafael

#23 lore raw
In reply to Jonathan Corbet
Quoted 16 lines · Jonathan Corbet

As your relatively new documentation subsystem maintainer, I think there might be reason to talk about docs for a bit at the summit.

Currently, we have two major branches to our docs. One, the bulk of Documentation/, is a haphazard collection of text files of varying quality and applicability to current kernels. The other is a set of more organized DocBook documents.

Despite their disorganized nature, the plain-text docs get the bulk of the attention. Anybody can manage to work with a text file. The DocBook stuff, instead, is kind of a pain. Working in DocBook itself is less than fun, few people dare to delve into the code that builds the docs, and the whole thing is rather fragile. Changing a struct to a union in the wireless subsystem a few months back broke the docs build, for example; few people test for such things and even fewer seem to know how to fix them.

...

Quoted 8 lines

I am not convinced that DocBook is really the right tool for the job here. It is complex, finicky, and it's a real pain to have to replace an entire laptop because you've worn out your shift key. It is, I believe, an impediment to improvements to our docs in general. If, instead, we used something more plain-text-like, I think life might get a lot easier; it could also enable the integration of all of our docs into something a bit more cohesive.

Yeah +1 from me on getting rid of DocBook.

I looked at it a bit in terms of asking people to write docs, and decided it was too painful to impose on people.

I know it's not *that* hard, but it's harder than it needs to be for the value it brings IMHO.

Quoted 7 lines

Markdown in one of its many forms might be a good alternative here. I'm also somewhat attracted by Sphinx, which is designed for documenting code already and could probably be made to work well with our existing kerneldoc comments without a whole lot of trouble. The Sphinx idea, though, is hobbled by the inconvenient fact that I've not had the time to develop it far enough to even have a vague idea of whether it would make real-world sense.

I'd vote for Markdown. It's very unobtrusive, reads nicely as plain text, and all the young kids know it from writing Github README.md files.

cheers

#24 lore raw

Hi Michael,

Quoted 10 lines
Quoted 7 lines

Markdown in one of its many forms might be a good alternative here. I'm also somewhat attracted by Sphinx, which is designed for documenting code already and could probably be made to work well with our existing kerneldoc comments without a whole lot of trouble. The Sphinx idea, though, is hobbled by the inconvenient fact that I've not had the time to develop it far enough to even have a vague idea of whether it would make real-world sense.

I'd vote for Markdown. It's very unobtrusive, reads nicely as plain text, and all the young kids know it from writing Github README.md files.

actually AsciiDoc might be something to look into as well.

Regards

Marcel

#25 lore raw

Hi

Am 4. August 2015 09:42:55 MESZ, schrieb Marcel Holtmann <[email protected]>:

Quoted 13 lines

Hi Michael,

Quoted 1 line
Quoted 1 line

Markdown in one of its many forms might be a good alternative here.

I'm

Quoted 1 line
Quoted 1 line

also somewhat attracted by Sphinx, which is designed for documenting

code

Quoted 1 line
Quoted 1 line

already and could probably be made to work well with our existing

kerneldoc

Quoted 1 line
Quoted 1 line

comments without a whole lot of trouble. The Sphinx idea, though,

is

Quoted 1 line
Quoted 1 line

hobbled by the inconvenient fact that I've not had the time to

develop it

Quoted 1 line
Quoted 1 line

far enough to even have a vague idea of whether it would make

Quoted 9 lines

real-world

Quoted 3 lines
Quoted 1 line

sense.

I'd vote for Markdown. It's very unobtrusive, reads nicely as plain

text, and

Quoted 1 line

all the young kids know it from writing Github README.md files.

actually AsciiDoc might be something to look into as well.

I'm also in for AsciiDoc, as it also allows easy generation of Pdfs and html files, with chapters and all that fancy stuff.

(Probably the reason why docbook was chosen back then, compared to the plaintext files)

Peter

Signature · 1 line
Sent from my mobile
#6 lore raw
In reply to Dmitry Torokhov
Quoted 5 lines · Dmitry Torokhov

My main issue with the DocBook is that it is not generated by default and it is actually takes a while to generate them; otherwise I woudl love to be able to catch at least issues breaking existing docs as early as possible.

0day does build docbook by default nowadays and screams at you if you break something. Well if the kerneldoc stuff can catch, like missing parameters in a function.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#31 lore raw
In reply to Jonathan Corbet
Quoted 14 lines · Jonathan Corbet

We have some interesting proposals for improvements on the DocBook side, including integrating Markdown support
(http://thread.gmane.org/gmane.linux.documentation/32140) and allowing embedded DocBook comments within structs
(https://lwn.net/Articles/653194/). I think they both have merit, but I hesitate to take them for a couple of reasons.

One is that changes in how docs are done would be imposed on everybody who is interested in keeping up the docs. They also can, in a sense, look like coding-style changes, which is an area where I fear to tread on my own. The other is that I wonder if we want to consider a bit of a more sweeping overhaul.

Quoted 7 lines

I am not convinced that DocBook is really the right tool for the job here. It is complex, finicky, and it's a real pain to have to replace an entire laptop because you've worn out your shift key. It is, I believe, an impediment to improvements to our docs in general. If, instead, we used something more plain-text-like, I think life might get a lot easier; it could also enable the integration of all of our docs into something a bit more cohesive.

The entire point of all the work Danilo's been doing is being able to pull out _all_ the actual text from the DocBook templates and move them into the code. The docbook template will simply be the scaffolding that pulls it all together and gives it the overall structure. That gives you:
- almost no xml to edit
- documentation as plain-text with markdown
- documentation always right next to the code

Especially the last one is crucial from my experience documenting the userspace side of the i915 driver. And the overall structure is also really important since a pile of linked chapters is a really good way for new people to dig into your docs. The pile of text files in Documentation imo fails badly at that, even if it would look prettier by using markdown, since if you don't know where to look you'll never find related bits of documentation. Crosslinking is key for useable docs imo, hence that's the first thing Danilo added. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#8 lore raw
In reply to Jonathan Corbet

Hi Jon,

Quoted 16 lines · On Monday 03 August 2015 08:33:11 Jonathan Corbet wrote:
Quoted 10 lines · On Mon, 03 Aug 2015 15:35:36 +0200 Rafael J. Wysocki wrote:
Quoted 6 lines

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

"thought" I guess?

I have to say I agree here.

Surely nobody thinks I was saying that the documentation-writing process can be automated! :) But we go to some lengths now to document our APIs in the code; I don't think we would want to break that.

Quoted 6 lines
Quoted 2 lines

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

My problem with inline documentation is that it makes easier for developers to write crappy doc and believe they've done their duty. It's not a technical issue, and I believe inline documentation has merits, especially for API documentation, but I've seen too many kerneldoc comments written as

/**
* int do_foo_bar(int foo, bool bar) - Do foo bar
* @foo: the foo for bar
* @bar: the value of bar
*
* This function does foo bar.
*
* Return 0 on success and a negative value on error. */

That's just useless. Worse, it can give the impression to reviewers that the function has been documented while it clearly hasn't. Maybe we just need to tighten the review process and push harder for documentation, at the risk of rejecting useful contributions. That's not a new problem though.

DocBook and kerneldoc documentation serve two very different purpose in my opinion. While kerneldoc mostly aims at documenting internal APIs, DocBook is more of a story telling kind of documentation, suitable for documenting subsystem architectures for instance. This could be moved to inline comments, but at the end of the day I don't think it will make a big difference. The reason why we miss quality "background" documentation today has in my opinion more to do with the fact that we don't require it.

A good (or bad example, depending on how you see it) of this is the DRM/KMS documentation available from Documentation/DocBook/drm.tmpl. I spent a considerable amount of time writing it, it got reviewed, merged in the kernel, and has now been rendered completely useless as the DRM/KMS internal API has evolved and the documentation hasn't been updated despite my repeated requests. That was extremely demotivating, and made me give up documenting DRM/KMS at all.

If we don't start treating documentation as equally valuable as code we will never fix the problem. I'd even argue that documentation should be treated as having more value than code.

One more point, DocBook does a good job at documenting userspace APIs,
especially ioctls. See http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-fmt.html for instance. Can Markdown or ReST achieve that ?

Signature · 3 lines
Regards,

Laurent Pinchart
#29 lore raw
In reply to Michael Ellerman

Hi Michael,

Quoted 29 lines · On Tuesday 04 August 2015 17:12:47 Michael Ellerman wrote:
Quoted 16 lines · Jonathan Corbet

As your relatively new documentation subsystem maintainer, I think there might be reason to talk about docs for a bit at the summit.

Currently, we have two major branches to our docs. One, the bulk of Documentation/, is a haphazard collection of text files of varying quality and applicability to current kernels. The other is a set of more organized DocBook documents.

Despite their disorganized nature, the plain-text docs get the bulk of the attention. Anybody can manage to work with a text file. The DocBook stuff, instead, is kind of a pain. Working in DocBook itself is less than fun, few people dare to delve into the code that builds the docs, and the whole thing is rather fragile. Changing a struct to a union in the wireless subsystem a few months back broke the docs build, for example; few people test for such things and even fewer seem to know how to fix them.

...

Quoted 8 lines

I am not convinced that DocBook is really the right tool for the job here. It is complex, finicky, and it's a real pain to have to replace an entire laptop because you've worn out your shift key. It is, I believe, an
impediment to improvements to our docs in general. If, instead, we used something more plain-text-like, I think life might get a lot easier; it could also enable the integration of all of our docs into something a bit more cohesive.

Quoted 4 lines

Yeah +1 from me on getting rid of DocBook.

I looked at it a bit in terms of asking people to write docs, and decided it was too painful to impose on people.

I don't buy that. The hard part in writing quality documentation is the documentation, not the format. Sure, we can make formating easier, but no matter how good the brushes you give him are, a kindergarten kid won't become Picasso overnight.

Quoted 11 lines

I know it's not *that* hard, but it's harder than it needs to be for the value it brings IMHO.

Quoted 7 lines

Markdown in one of its many forms might be a good alternative here. I'm also somewhat attracted by Sphinx, which is designed for documenting code already and could probably be made to work well with our existing kerneldoc comments without a whole lot of trouble. The Sphinx idea, though, is hobbled by the inconvenient fact that I've not had the time to develop it far enough to even have a vague idea of whether it would make real-world sense.

Quoted 2 lines

I'd vote for Markdown. It's very unobtrusive, reads nicely as plain text, and all the young kids know it from writing Github README.md files.

How can we document ioctls such as in http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-fmt.html using Markdown ?

Signature · 3 lines
Regards,

Laurent Pinchart
#9 lore raw
In reply to Laurent Pinchart
Quoted 41 lines · Laurent Pinchart

Hi Jon,

Quoted 16 lines · On Monday 03 August 2015 08:33:11 Jonathan Corbet wrote:
Quoted 10 lines · On Mon, 03 Aug 2015 15:35:36 +0200 Rafael J. Wysocki wrote:
Quoted 6 lines

While I don't discourage it, I am not a fan of automated documentation. As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

"thought" I guess?

I have to say I agree here.

Surely nobody thinks I was saying that the documentation-writing process can be automated! :) But we go to some lengths now to document our APIs in the code; I don't think we would want to break that.

Quoted 6 lines
Quoted 2 lines

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

My problem with inline documentation is that it makes easier for developers to write crappy doc and believe they've done their duty. It's not a technical issue, and I believe inline documentation has merits, especially for API documentation, but I've seen too many kerneldoc comments written as

/**
* int do_foo_bar(int foo, bool bar) - Do foo bar
* @foo: the foo for bar
* @bar: the value of bar
*
* This function does foo bar.
*
* Return 0 on success and a negative value on error. */

Quoted 4 lines

That's just useless. Worse, it can give the impression to reviewers that the function has been documented while it clearly hasn't. Maybe we just need to tighten the review process and push harder for documentation, at the risk of rejecting useful contributions. That's not a new problem though.

Yeah api/struct docs alone aren't enough, that's why you need overview sections to tie it all together. And in those you really need links to all the various entry points to be able to digg out the details. Same goes for api docs themselves, I try to always list related functions in there so that you can easily jump to more specific functions or between init/free and similar pairs.

Quoted 7 lines

DocBook and kerneldoc documentation serve two very different purpose in my opinion. While kerneldoc mostly aims at documenting internal APIs, DocBook is more of a story telling kind of documentation, suitable for documenting subsystem architectures for instance. This could be moved to inline comments, but at the end of the day I don't think it will make a big difference. The reason why we miss quality "background" documentation today has in my opinion more to do with the fact that we don't require it.

Yeah pure function docs alone are useless, which is why I always ask for an intro/overview section.

Quoted 7 lines

A good (or bad example, depending on how you see it) of this is the DRM/KMS documentation available from Documentation/DocBook/drm.tmpl. I spent a considerable amount of time writing it, it got reviewed, merged in the kernel, and has now been rendered completely useless as the DRM/KMS internal API has evolved and the documentation hasn't been updated despite my repeated requests. That was extremely demotivating, and made me give up documenting DRM/KMS at all.

drm having gone downhill is why I want to pull it out of the template into code. E.g. we have extensive docs for vtables, but since they're not next to the vtables in the code no one updates those. And duplicating a summary in kerneldoc and then the extended version in the docbook is even worse imo.

Also editing docbook is really a major pain imo. The drm docbook file is currently at over 4k lines, and that's just not something you can manage any more sensibly.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#30 lore raw
In reply to Laurent Pinchart
Quoted 9 lines · Laurent Pinchart
Quoted 4 lines

Yeah +1 from me on getting rid of DocBook.

I looked at it a bit in terms of asking people to write docs, and decided it was too painful to impose on people.

I don't buy that. The hard part in writing quality documentation is the documentation, not the format. Sure, we can make formating easier, but no matter how good the brushes you give him are, a kindergarten kid won't become Picasso overnight.

Yeah let's please not just throw out tooling because we don't like the color of the bikeshed. Afaik all the code documentation tools are incompatible in slight ways, which means any change means someone gets to fix up tons of little issues all over. And if you don't do that and just add yet another doc system then it'll be even worse.

And like Laurent said presentation for docs is fairly irrelevant, it's the content that matters. And as long as you can do basic doc structure, linking and have a bit of markup you're good to go. And there's very little we imo need to add to the existing kerneldoc/docbook-template toolchain to get there. The patches are also already written ...
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#12 lore raw
In reply to Laurent Pinchart

If you write your documentation in docbook, I can promise you that I have never read it.

I have all the tools to read it. I remember this because it requires 700MB of dependencies. I was downloading it to read the kdbus documentation. I remember this. I don't think I read the kdbus documentation in the end. I certainly don't know, without googling, how to use build the docbook output.

Presumably, someone has a webpage with all the docs pre-built. I looked for your drm.pdf file on google. The first link was
http://kernelnewbies.org/Documents/Kernel-Docbooks
which has some pre-built docs but they are out of date. The next link
was: http://billauer.co.il/blog/2014/02/docbook-html-pdf-rtf/ where the guy also can't figure out how to build his documention. It's like he actually wanted to read it (which is so rare) but he wasn't able to figure it out.

regards,
dan carpenter

#13 lore raw
Quoted 5 lines · Dan Carpenter

I have all the tools to read it. I remember this because it requires 700MB of dependencies. I was downloading it to read the kdbus documentation. I remember this. I don't think I read the kdbus documentation in the end. I certainly don't know, without googling, how to use build the docbook output.

I gave up trying to build the docbook docs many years ago.

Cross-compiling kernels for niche architectures is peanuts compared to generating docbook output ;-(

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#10 lore raw
In reply to Daniel Vetter

Hi Daniel,

Quoted 78 lines · On Tuesday 04 August 2015 15:03:04 Daniel Vetter wrote:
Quoted 44 lines · Laurent Pinchart
Quoted 18 lines · On Monday 03 August 2015 08:33:11 Jonathan Corbet wrote:
Quoted 9 lines · On Mon, 03 Aug 2015 15:35:36 +0200 Rafael J. Wysocki wrote:
Quoted 8 lines

While I don't discourage it, I am not a fan of automated documentation.
As you and mtk would know, writing high quality, informative, systems software documentation is an involved process. And it should be, imo. Same goes for describing APIs and algorithms in code comments. Sure, automation has its pros, particularly keeping docs up to date; yet this does not outweigh a well crafted document, which involves actual though.

Quoted 3 lines

"thought" I guess?

I have to say I agree here.

Surely nobody thinks I was saying that the documentation-writing process can be automated! :) But we go to some lengths now to document our APIs in the code; I don't think we would want to break that.

Quoted 7 lines
Quoted 3 lines

Not to mention the fact that if you are browsing the kernel tree via a web frontend or LXR, for example, plain text docs are really good to have.

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

My problem with inline documentation is that it makes easier for developers to write crappy doc and believe they've done their duty. It's not a technical issue, and I believe inline documentation has merits, especially for API documentation, but I've seen too many kerneldoc comments written as

/**

* int do_foo_bar(int foo, bool bar) - Do foo bar
* @foo: the foo for bar
* @bar: the value of bar
*
* This function does foo bar.
*
* Return 0 on success and a negative value on error. */

Quoted 5 lines

That's just useless. Worse, it can give the impression to reviewers that the function has been documented while it clearly hasn't. Maybe we just need to tighten the review process and push harder for documentation, at the risk of rejecting useful contributions. That's not a new problem though.

Yeah api/struct docs alone aren't enough, that's why you need overview sections to tie it all together. And in those you really need links to all the various entry points to be able to digg out the details. Same goes for api docs themselves, I try to always list related functions in there so that you can easily jump to more specific functions or between init/free and similar pairs.

Quoted 8 lines

DocBook and kerneldoc documentation serve two very different purpose in my opinion. While kerneldoc mostly aims at documenting internal APIs, DocBook is more of a story telling kind of documentation, suitable for documenting subsystem architectures for instance. This could be moved to inline comments, but at the end of the day I don't think it will make a big difference. The reason why we miss quality "background" documentation today has in my opinion more to do with the fact that we don't require it.

Yeah pure function docs alone are useless, which is why I always ask for an intro/overview section.

Quoted 7 lines

A good (or bad example, depending on how you see it) of this is the DRM/KMS documentation available from Documentation/DocBook/drm.tmpl. I spent a considerable amount of time writing it, it got reviewed, merged in the kernel, and has now been rendered completely useless as the DRM/KMS internal API has evolved and the documentation hasn't been updated despite my repeated requests. That was extremely demotivating, and made me give up documenting DRM/KMS at all.

Quoted 5 lines

drm having gone downhill is why I want to pull it out of the template into code. E.g. we have extensive docs for vtables, but since they're not next to the vtables in the code no one updates those. And duplicating a summary in kerneldoc and then the extended version in the docbook is even worse imo.

The problem is two-fold. Extensive vtables documentation is definitely good, but it mostly serves as API reference documentation. I believe we also need a guide type of documentation that goes through the bits and pieces drivers need in a logical order. That should of course be linked to the vtables reference documentation, but it should be a natural text flow type of documentation that driver developers can read from start to end, guiding them in the process of writing a DRM/KMS driver.

Quoted 4 lines

Also editing docbook is really a major pain imo. The drm docbook file is currently at over 4k lines, and that's just not something you can manage any more sensibly.
-Daniel

Signature · 3 lines
Regards,

Laurent Pinchart
#14 lore raw
In reply to Geert Uytterhoeven
Quoted 11 lines · Geert Uytterhoeven
Quoted 5 lines · Dan Carpenter

I have all the tools to read it. I remember this because it requires 700MB of dependencies. I was downloading it to read the kdbus documentation. I remember this. I don't think I read the kdbus documentation in the end. I certainly don't know, without googling, how to use build the docbook output.

I gave up trying to build the docbook docs many years ago.

Cross-compiling kernels for niche architectures is peanuts compared to generating docbook output ;-(

Don't build the pdf, it's useless. Build the html pages with all the links and stuff, and that just about requires an xlstproc plus the docbook stylesheets. Building the html pages is as simple as

$ make htmldocs

and make help will tell you that too.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#15 lore raw
In reply to Geert Uytterhoeven
Quoted 11 lines · On Tuesday 04 August 2015 16:05:16 Geert Uytterhoeven wrote:
Quoted 5 lines · Dan Carpenter

I have all the tools to read it. I remember this because it requires 700MB of dependencies. I was downloading it to read the kdbus documentation. I remember this. I don't think I read the kdbus documentation in the end. I certainly don't know, without googling, how to use build the docbook output.

I gave up trying to build the docbook docs many years ago.

Cross-compiling kernels for niche architectures is peanuts compared to generating docbook output ;-(

That's interesting, I haven't had trouble compiling the DocBook documentation to HTML for several years now (I do it every time I have to patch the V4L2 or DRM documentation). Compiling to PDF is another story though.

Signature · 3 lines
Regards,

Laurent Pinchart
#11 lore raw
In reply to Laurent Pinchart
Quoted 14 lines · Laurent Pinchart
Quoted 8 lines
Quoted 7 lines

A good (or bad example, depending on how you see it) of this is the DRM/KMS documentation available from Documentation/DocBook/drm.tmpl. I spent a considerable amount of time writing it, it got reviewed, merged in the kernel, and has now been rendered completely useless as the DRM/KMS internal API has evolved and the documentation hasn't been updated despite my repeated requests. That was extremely demotivating, and made me give up documenting DRM/KMS at all.

Quoted 5 lines

drm having gone downhill is why I want to pull it out of the template into code. E.g. we have extensive docs for vtables, but since they're not next to the vtables in the code no one updates those. And duplicating a summary in kerneldoc and then the extended version in the docbook is even worse imo.

Quoted 7 lines

The problem is two-fold. Extensive vtables documentation is definitely good, but it mostly serves as API reference documentation. I believe we also need a guide type of documentation that goes through the bits and pieces drivers need in a logical order. That should of course be linked to the vtables reference documentation, but it should be a natural text flow type of documentation that driver developers can read from start to end, guiding them in the process of writing a DRM/KMS driver.

Oh the vtables is just the blocker why I didn't start this yet - I really don't like the current duplication we have. Fully agreed that we need overview docs to tie it all together and explain how it works. With markdown we can even do inline code snippets, which should help a lot with explaining how to use certain helper libraries. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#20 lore raw
In reply to Laurent Pinchart
Quoted 10 lines · Laurent Pinchart
Quoted 3 lines · On Monday 03 August 2015 08:33:11 Jonathan Corbet wrote:

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

My problem with inline documentation is that it makes easier for developers to write crappy doc and believe they've done their duty. It's not a technical issue, and I believe inline documentation has merits, especially for API documentation, but I've seen too many kerneldoc comments written as

That's not really anything to do with the fact that the documentation is inline, it's more to do with documentation that people are forced to write without any obvious interest being shown in the results. Anything else that's done as a pure checkbox effort will have similar effects.

Quoted 4 lines

That's just useless. Worse, it can give the impression to reviewers that the function has been documented while it clearly hasn't. Maybe we just need to tighten the review process and push harder for documentation, at the risk of rejecting useful contributions. That's not a new problem though.

I'm not sure that more requirements for documentation will help that much, there's a definite skill in writing and review documentation which isn't all that common. Insisting on people writing more without doing something to address that may just result in more bad documentation, but of course it's a really hard problem to address :(

Quoted 3 lines

If we don't start treating documentation as equally valuable as code we will never fix the problem. I'd even argue that documentation should be treated as having more value than code.

The documentation I tend to read is commit logs, those we do tend to value much more and it shows.

#16 lore raw
In reply to Laurent Pinchart
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Tuesday 04 August 2015 16:05:16 Geert Uytterhoeven wrote:
Quoted 5 lines · Dan Carpenter

I have all the tools to read it. I remember this because it requires 700MB of dependencies. I was downloading it to read the kdbus documentation. I remember this. I don't think I read the kdbus documentation in the end. I certainly don't know, without googling, how to use build the docbook output.

Quoted 8 lines
Quoted 4 lines

I gave up trying to build the docbook docs many years ago.

Cross-compiling kernels for niche architectures is peanuts compared to generating docbook output ;-(

That's interesting, I haven't had trouble compiling the DocBook documentation to HTML for several years now (I do it every time I have to patch the V4L2 or DRM documentation). Compiling to PDF is another story though.

Right.

I've just tried "make htmldocs", and it indeed works (after installing one quite obvious dependency).
"make pdfdocs" is stuck in:

) (/usr/share/texlive/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty (/usr/share/texlive/texmf-dist/tex/latex/oberdiek/grfext.sty) ! Undefined control sequence.
<recently read> \AppendGraphicsExtensions

l.451 \fi

?
! Emergency stop.
<recently read> \AppendGraphicsExtensions

l.451 \fi

! ==> Fatal error occurred, no output PDF file produced!

which a quick Google search didn't manage to provide a solution for.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#22 lore raw
In reply to Laurent Pinchart
Quoted 3 lines · On Tue, 04 Aug 2015 15:50:33 +0300 Laurent Pinchart…

Maybe we just need to
tighten the review process and push harder for documentation, at the risk of rejecting useful contributions. That's not a new problem though.

It's so not-new that this idea has come up at a few kernel summits in the past. But better documentation is not something we can wish into existence; it's like saying we'll tighten the process to require more code review.

That's why "how do we get more documentation?" wasn't really the question I posed at the beginning. We get it the way we get more code: get companies to pay people to write it. I don't think we can really make that happen at the kernel summit.

We *can* maybe think about how we want our tools to look to make the process of writing and maintaining documentation easier, though. Tools won't solve the problem either, but maybe we can at least reduce the impedance somewhat.

jon

#18 lore raw
In reply to Konstantin Ryabitsev
Quoted 4 lines · Konstantin Ryabitsev
Quoted 1 line · Dan Carpenter

Presumably, someone has a webpage with all the docs pre-built.

https://www.kernel.org/doc/htmldocs/

That's good. When this thread started I tried looking for an online instance of prebuilt docs, but the latest one I found was for 2.6.30.

If this is maintained consistently, maybe this site should be (wait for it...) documented somewhere. ;-)

On a more serious note, I'd be in favor of:
1) switching to markdown or AsciiDoc. Pandoc markdown looks pretty capable, but I wouldn't want to introduce a dependency on Haskell. I think for the types of things we do, just about any markdown would do. Pandoc claims to be able to convert between many different formats, including AsciiDoc, and to convert to many different outputs.

I think having plain text will lower the barrier to editing the docs.

2) keeping the intro texts in the source files. That way doc changes should be more visible to maintainers, and they're arguably more accessible when you're poking around the code.
-- Tim

#19 lore raw

Hi Tim,

Quoted 21 lines · On Tuesday 04 August 2015 11:21:35 Tim Bird wrote:
Quoted 4 lines · Konstantin Ryabitsev
Quoted 1 line · Dan Carpenter

Presumably, someone has a webpage with all the docs pre-built.

https://www.kernel.org/doc/htmldocs/

That's good. When this thread started I tried looking for an online instance of prebuilt docs, but the latest one I found was for 2.6.30.

If this is maintained consistently, maybe this site should be (wait for it...) documented somewhere. ;-)

On a more serious note, I'd be in favor of:
1) switching to markdown or AsciiDoc. Pandoc markdown looks pretty capable, but I wouldn't want to introduce a dependency on Haskell. I think for the types of things we do, just about any markdown would do. Pandoc claims to be able to convert between many different formats, including AsciiDoc, and to convert to many different outputs.

Quoted 7 lines

I think having plain text will lower the barrier to editing the docs.

2) keeping the intro texts in the source files. That way doc changes should be more visible to maintainers, and they're arguably more accessible when you're poking around the code.

It's more than intro text in my opinion. Look at
https://www.kernel.org/doc/htmldocs/drm/index.html, it tries to be a guide/walk-through type of documentation. The amount of documentation is so large that it would be quite impractical to keep it in source files. Or we could have source files containing only documentation comments, but that would defeat the point :-)

Signature · 3 lines
Regards,

Laurent Pinchart
#21 lore raw
In reply to Mark Brown
Quoted 22 lines · Mark Brown
Quoted 10 lines · Laurent Pinchart
Quoted 3 lines · On Monday 03 August 2015 08:33:11 Jonathan Corbet wrote:

The nice thing about formats like Markdown or ReST is that they *are* plain text for all practical purposes. Much better than DocBook in that regard.

My problem with inline documentation is that it makes easier for developers to write crappy doc and believe they've done their duty. It's not a technical issue, and I believe inline documentation has merits, especially for API documentation, but I've seen too many kerneldoc comments written as

That's not really anything to do with the fact that the documentation is inline, it's more to do with documentation that people are forced to write without any obvious interest being shown in the results. Anything else that's done as a pure checkbox effort will have similar effects.

Quoted 4 lines

That's just useless. Worse, it can give the impression to reviewers that the function has been documented while it clearly hasn't. Maybe we just need to tighten the review process and push harder for documentation, at the risk of rejecting useful contributions. That's not a new problem though.

Quoted 5 lines

I'm not sure that more requirements for documentation will help that much, there's a definite skill in writing and review documentation which isn't all that common. Insisting on people writing more without doing something to address that may just result in more bad documentation, but of course it's a really hard problem to address :(

While it's clear that, like skill at coding, skill at documentation is wildly variable, almost everyone gets better at it with practice! Like most problems, the problem would be more tractable if people were paid to work on this. For a short while, there was an LF documentation fellowship that had a goal to help things here. It had variable success. For a some months it supported me (as the second holder of the fellowship) to work full time on man pages, but there was so much man pages work, that kernel docs didn't get too much of a look in. And then a few months later (2008 crash), the funds went away. I raised the topic of reanimating that fellowship a couple of times subsequently, but nothing came of it. That was however quite a few years ago. Given recent initiatives from LF (such as the Critical Infrastructure Initiative), I wonder whether it's worth someone checking in with LF about the possibility of getting this going again. One thing I'd caution against though is the idea that someone can just be paid to write kernel documentation. Many parts of the kernel require significant time just to grok what's going on, let alone document it. A much more efficient route to producing good documentation would I think be to have someone with solid writing skills (as well as coding skills) support others (i.e., subsystem developers and maintainers) to produce documentation, by providing writing assistance, technical review, editing, and so on. Cheers,

Michael

Signature · 3 lines
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
#26 lore raw
In reply to Peter Huewe
Quoted 25 lines · Peter Huewe

Hi

Am 4. August 2015 09:42:55 MESZ, schrieb Marcel Holtmann <[email protected]>:

Quoted 13 lines

Hi Michael,

Quoted 1 line
Quoted 1 line

Markdown in one of its many forms might be a good alternative here.

I'm

Quoted 1 line
Quoted 1 line

also somewhat attracted by Sphinx, which is designed for documenting

code

Quoted 1 line
Quoted 1 line

already and could probably be made to work well with our existing

kerneldoc

Quoted 1 line
Quoted 1 line

comments without a whole lot of trouble. The Sphinx idea, though,

is

Quoted 1 line
Quoted 1 line

hobbled by the inconvenient fact that I've not had the time to

develop it

Quoted 1 line
Quoted 1 line

far enough to even have a vague idea of whether it would make

Quoted 9 lines

real-world

Quoted 3 lines
Quoted 1 line

sense.

I'd vote for Markdown. It's very unobtrusive, reads nicely as plain

text, and

Quoted 1 line

all the young kids know it from writing Github README.md files.

actually AsciiDoc might be something to look into as well.

Quoted 7 lines

I'm also in for AsciiDoc, as it also allows easy generation of Pdfs and html files, with chapters and all that fancy stuff.

(Probably the reason why docbook was chosen back then, compared to the plaintext files)

From a man-pages point of view, the discussion of tooling is interesting.

For some years now I've contemplated a switch from groff's man macros, since some people do not like to work with them (although the basics are pretty simple, IMO). If there was a concerted movement to Asciidoc or Markdown for kernel docs, I'd be willing to invest some effort in the feasibility of switching man-pages to the same format. I'm not knowledgeable in either of those tools, but at a first glance, Asciidoc seems more interesting, since it seems to embrace a wider variety of output formats, in particular, PDF and Linux man-page macros (and also TeX). It doesn't appear that Markdown can do that. I know even less about Sphinx, although its output capabilities, as described at http://sphinx-doc.org/index.html , seem interesting also.

Cheers,

Michael

Signature · 3 lines
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
#27 lore raw
Quoted 4 lines · Michael Kerrisk (man-pages)

From a man-pages point of view, the discussion of tooling is interesting. For some years now I've contemplated a switch from groff's man macros, since some people do not like to work with them (although the basics are pretty simple, IMO). If there was a concerted movement to Asciidoc or

Quoted 8 lines

Markdown for kernel docs, I'd be willing to invest some effort in the feasibility of switching man-pages to the same format. I'm not knowledgeable in either of those tools, but at a first glance, Asciidoc seems more interesting, since it seems to embrace a wider variety of output formats, in particular, PDF and Linux man-page macros (and also TeX). It doesn't appear that Markdown can do that. I know even less about Sphinx, although its output capabilities, as described at http://sphinx-doc.org/index.html , seem interesting also.

I've seen multiple scripts to convert mdwn to groff for manual pages. And once you have groff, you can typeset PDFs from that, right?

The biggest thing I've noticed about asciidoc is that it seems to be incredibly *slow*. For all the projects I know of that use it, building the documentation is by far the slowest part of the build process.

Honestly, groff isn't *that* hard to deal with.

- Josh Triplett

#28 lore raw
In reply to Michael Kerrisk (man-pages)
Quoted 2 lines · Michael Kerrisk (man-pages)

I know even less about Sphinx, although its output capabilities, as described at http://sphinx-doc.org/index.html , seem interesting also.

Markdown has vastly simpler syntax than ReST, which has the upside of being easier to learn, and the downside of being quite limited. I mean, it doesn't even do tables unless you use an extended flavour (GitHub-flavoured MarkDown).

For this reason I would recommend ReST over MD.

I find rst2man to be a great way to quickly write decent manpages. E.g.:

https://raw.githubusercontent.com/mricon/grokmirror/master/man/grok-pull.1.rst

Which generates the following man page:

https://raw.githubusercontent.com/mricon/grokmirror/master/man/grok-pull.1

Many will attest that the .rst source is a lot easier on the eyes than final groff.

Hope this helps.

Best,

Signature · 3 lines
Konstantin Ryabitsev
Linux Foundation Collab Projects
Montréal, Québec