uboot-doc User's Manual Generation Tool

I've been working on writing a User's Manual generation tool, and before I get too far I'd like to see if this is something the U-Boot community would be interested in.

General Control Flow:
C pre-processor -> doc_stream.pl : pre-processed code gets everything but comments with an @ stripped away
doc_stream.pl -> autoconf_doc.txt : "interesting" comments get sent to this file
autoconf_doc.txt -> uboot-doc : file is parsed and comments are turned to XML
*.tmpl -> uboot-doc : template files are processed into XML *.xml -> xsltproc : XML is converted into desired output format *.fo -> fop : (PDFs only) fo files are translated to PDF

I see several advantages to adopting this scheme.
- Documentation should be easier to keep in sync with code

- DocBook XML output can be used to generate webpages, PDFs, text, etc, which are all extensible

- People don't have to reinvent the wheel when writing documentation

- Source code ends up being thouroughly commented

These patches are just meant to be an example of how in-code documentation could be used. You'll also notice there are many warnings regarding variables the manual is referencing which aren't defined yet. I wanted to get some feedback before diving in too far. What do others think? Is this worth pursuing?

Dear John Schmoller,

Quoted 4 lines · In message <[email protected]> you wrote:

I've been working on writing a User's Manual generation tool, and before I get too far I'd like to see if this is something the U-Boot community would be interested in.

Anything wrong with the User's Manual generation tool we already have in use (DUTS) for example to generate the DULG?

Quoted 10 lines

General Control Flow:
C pre-processor -> doc_stream.pl : pre-processed code gets everything but comments with an @ stripped away
doc_stream.pl -> autoconf_doc.txt : "interesting" comments get sent to this file
autoconf_doc.txt -> uboot-doc : file is parsed and comments are turned to XML
*.tmpl -> uboot-doc : template files are processed into XML *.xml -> xsltproc : XML is converted into desired output format *.fo -> fop : (PDFs only) fo files are translated to PDF

What you describe does not sound like a user's manual to me, but more like some API documentation - whioch is a completely different thing (and something we really don't have yet).

Your approach may be suitable for standard documents, but in many years of working with U-Boot (and Linux) we found that it does not work so well with the specific needs we have for a User's Manual. One issue is that you have to support many different boards (well, maybe not you as a user, but we as a community). And you have to include examples. And examples must really fit the board. If you for example provide a manual entry for the "erase" command you better make sure that your example does not erase a range of flash that on some board happens to hold the U-Boot image. etc.

That's how we came to the DULG we have today. I'm still convinced that this is a really well-working approach.

It seems you don;t address this issue yet.

Quoted 3 lines

I see several advantages to adopting this scheme.
- Documentation should be easier to keep in sync with code

Better than what we have with the DULG? I doubt it.

Quoted 2 lines

- DocBook XML output can be used to generate webpages, PDFs, text, etc, which are all extensible

We do the same with the DULG.

Quoted 1 line

- People don't have to reinvent the wheel when writing documentation

Well, you just did that, it seems ;-)

Quoted 7 lines

- Source code ends up being thouroughly commented

These patches are just meant to be an example of how in-code documentation could be used. You'll also notice there are many warnings regarding variables the manual is referencing which aren't defined yet. I wanted to get some feedback before diving in too far. What do others think? Is this worth pursuing?

Hm... we cannot look at your patches, you just posted the patch statistics, but no content.

The documentation itself seems to duplicate a lot of content we have in the DULG. I have to admit that I'd prefer to see the effort invested in improving the existing manual, than to come up with something different (and, as it seems, less flexible).

It seems to me that DUTS is designed to test U-Boot and also automates the running of commands whose output can be put online in the DULG. I

didn't notice any documented procedure on how to turn the DULG into XML, extensible PDFs, etc. It also seems as if the DULG is a combination of hand-edited wiki pages as well as the DUTS output? I was hoping to

develop a system that's completely automated. I also noticed in a quick

probe around that a few items in the DULG seem to be out of sync (imd vs. i2c md, source vs. autoscr, etc.) and DHCP seems to be out of date as well. Is the process for updating the DULG automatic? If so, how is it done?

As I mentioned, I borrowed this idea from the kernel-doc script in Linux, which does do API documentation. But my hope for the uboot-doc tool would be to create user documentation, or a manual we'd provide to a customer when they purchased a product that would describe available commands, environment variables, common operations, etc.

Quoted 9 lines

Your approach may be suitable for standard documents, but in many years of working with U-Boot (and Linux) we found that it does not work so well with the specific needs we have for a User's Manual. One issue is that you have to support many different boards (well, maybe not you as a user, but we as a community). And you have to include examples. And examples must really fit the board. If you for example provide a manual entry for the "erase" command you better make sure that your example does not erase a range of flash that on some board happens to hold the U-Boot image. etc.

It's my hope that the documentation system I proposed can be made flexible enough to support things such as this without too much headache. That's the hope at least :)

This would make DUTS/DULG more appealing. What is the process of generating DocBook XML from DUTS/DULG?

It's true, I may have. :) On the other hand, it seems that there are still a lot of people who are in the same boat. Most manuals I have seen from other embedded companies (Freescale, Analog Devices, etc.) seem to provide their own format/content. Additionally, most companies will prefer to have their content formatted to match the rest of their manuals, which is easily done from DocBook XML. Again, if you can do that with DUTS/DULG, then that's great and probably eliminates the need for this tool.

Quoted 2 lines

Hm... we cannot look at your patches, you just posted the patch statistics, but no content.

This is just the patch series introduction, the actual content appears in 1-3.

As I mentioned, I'm not all that familiar with the abilities of DUTS/DULG, but I got the impression that there was still a fair amount of manual labor involved for each manual. Thus most people (that I'm

I was hoping that creating documentation from comments in the source would be easier to maintain and providing DocBook output would allow others to more easily reuse the U-Boot manual contents.

Dear John,

Quoted 3 lines · in message <1248813631.3915.102.camel@johns> you wrote:

It seems to me that DUTS is designed to test U-Boot and also automates the running of commands whose output can be put online in the DULG. I

Correct. And the DULG itself is a wiki (TWiki at the moment, to be converted to Foswiki ASAP) based framework which holds the "static" parts of the documentation.

On Tue 28 Jul 2009 17:27, Wolfgang Denk pondered:

Quoted 10 lines

Dear John,

Quoted 3 lines · in message <1248813631.3915.102.camel@johns> you wrote:

It seems to me that DUTS is designed to test U-Boot and also automates the running of commands whose output can be put online in the DULG. I

Correct. And the DULG itself is a wiki (TWiki at the moment, to be converted to Foswiki ASAP) based framework which holds the "static" parts of the documentation.

While we are on the topics of wikis, and U-Boot docs, can we discuss the license?

http://www.denx.de/wiki/view/DULG/Introduction#Section_2.1.

Since it is "is not permitted to sell this document or the derivative work or to include it into any package or distribution that is not freely available to everybody."

Not meaning to interpret licensing, but to me that means I can't copy/paste sections into end product documentation, and ship the product for a fee...

Was that the intent?

Quoted 2 lines

Not meaning to interpret licensing, but to me that means I can't copy/paste sections into end product documentation, and ship the product for a fee...

That's how it is currently, yes.

Quoted 1 line

Was that the intent?

Not really - but I'll better let Wolfgang answer that question as he came up with the original licence text.

It is on my priority list however to assemble a list of all contributors and ask them if they agree to relicence their contributions under the FDL.

Hi,

In message <[email protected]> Detlev Zundel wrote:
...

Quoted 10 lines

Quoted 3 lines

Not meaning to interpret licensing, but to me that means I can't copy/paste sections into end product documentation, and ship the product for a fee...

That's how it is currently, yes.

Quoted 1 line

Was that the intent?

Not really - but I'll better let Wolfgang answer that question as he came up with the original licence text.

Originally this actualy has been my intent. But that was a long, long time ago, back when I was writing all these documentation only by myself.

Quoted 3 lines

It is on my priority list however to assemble a list of all contributors and ask them if they agree to relicence their contributions under the FDL.

Consider my permission as granted :-)

I assume - no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts? or did you desire something else?

Since Front-Cover Text is limited to 5 words, and Back-Cover Text is limited to 25 words - how about (as Back-Cover Text) "This manual includes sections from the Das U-Boot documentation, which can be found at
http://www.denx.de/wiki/U-Boot and is copyright it's authors. Included under the Free Documentation License (v1.3)"

?

Dear Robin Getz,

Quoted 3 lines · In message <[email protected]> you wrote:

I assume - no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts? or did you desire something else?

We don't have such detailed plans yet.

Quoted 15 lines

Hi Mike,

Quoted 13 lines · Detlev Zundel

Quoted 11 lines

Quoted 4 lines

we maintain a Blackfin-specific u-boot wiki that goes into quite a bit of detail, some of which is duplicated with the main u-boot wiki. how do people feel about extending the u-boot wiki to allow for arch-specific details ?

What exactly do you have in mind? I surely don't see any principal problem here.

It would certainly be valuable to get all U-Boot related info collected in a central place and have pointers wherever that make sense...

Quoted 6 lines

Quoted 4 lines

from my reading of the wiki, it's more of a technical/command reference than a guide. the wiki we maintain is geared to be more of a guide. i think the two can be merged, i just dont want to convert things only to find out people dont want to take it that direction.

Just to be clear, we are discussing the DULG wiki, right?

is there any other worth talking about :)

Quoted 4 lines

I agree that in the current state the documentation is more a reference but IIRC that wasn't really a conscious design decision. It simply turned out this way in the end.

Quoted 3 lines

So I do not see any general problem in adding "guide style" sections in there. Maybe then most of the current documentation can then be shifted to a "commands reference" section.

OK

Quoted 5 lines

One problem I see though is how to correctly adapt such sections to the board specific nature of the DULG. Hopefully we can get away with mostly generic text passages and only a few ifdefs. It would be very helpful to know more concrete plans (outline!) to think further about these implications.

It is on my priority list however to assemble a list of all contributors and ask them if they agree to relicence their contributions under the FDL.

Dear John,

Quoted 3 lines · in message <1248813631.3915.102.camel@johns> you wrote:

It seems to me that DUTS is designed to test U-Boot and also automates the running of commands whose output can be put online in the DULG. I

Correct. And the DULG itself is a wiki (TWiki at the moment, to be converted to Foswiki ASAP) based framework which holds the "static" parts of the documentation.

Quoted 3 lines

didn't notice any documented procedure on how to turn the DULG into XML, extensible PDFs, etc. It also seems as if the DULG is a combination of hand-edited wiki pages as well as the DUTS output? I was hoping to

You probably can export the DULG into XML, too - but what would that be good for? At the moment we export it into PDF, PostScript, HTML and plain text.

Quoted 1 line

develop a system that's completely automated. I also noticed in a quick

Hm... "completely automated" is a nice buzzword, but you still have to write the documentation in the first place.

The nice thing of the wiki based approach is that you can have a (even random) collection of pages which can be "linearized" and brought into arbitrary sets of linear doucumentation - this is what the WebOrder pages are good for - see
http://www.denx.de/wiki/DULG/WebOrder

So you can use the same set of wiki pages an genreate for example the full fledged manual, a short version including only the most significant topics and an extended version including other stuff that is normally not part of the manual - all from the same envrionment.

Quoted 4 lines

probe around that a few items in the DULG seem to be out of sync (imd vs. i2c md, source vs. autoscr, etc.) and DHCP seems to be out of date as well. Is the process for updating the DULG automatic? If so, how is it done?

Soemone still has to write the documentation - and this is not always

done  in  sync  with  the  code.  Your  approach  of  including   the
documentation  with  the  source  code  makes it more likely that one

remebers to update the docs as well, but just remember how many places there are where code and comments don't agree - it's no guarantee either.

If you find such areas in the DULG that are out of sync or missing please feel free to add them - everybody can contribute and modify the pages or add new content. And everyboidy can modify and extend the DUTS test cases, too.

Quoted 5 lines

As I mentioned, I borrowed this idea from the kernel-doc script in Linux, which does do API documentation. But my hope for the uboot-doc tool would be to create user documentation, or a manual we'd provide to a customer when they purchased a product that would describe available commands, environment variables, common operations, etc.

That's what the DULG does, so you are truely duplicating efforts.

Quoted 10 lines

Quoted 9 lines

Your approach may be suitable for standard documents, but in many years of working with U-Boot (and Linux) we found that it does not work so well with the specific needs we have for a User's Manual. One issue is that you have to support many different boards (well, maybe not you as a user, but we as a community). And you have to include examples. And examples must really fit the board. If you for example provide a manual entry for the "erase" command you better make sure that your example does not erase a range of flash that on some board happens to hold the U-Boot image. etc.

Quoted 3 lines

It's my hope that the documentation system I proposed can be made flexible enough to support things such as this without too much headache. That's the hope at least :)

Heh. Look at the DULG implementation... it's simple for the first 3 or 4 boards, especially when these are similar. But try supporting ARM and PowerPC and MIPS boards from N different vendors, with different kernel versions etc.

Quoted 2 lines

This would make DUTS/DULG more appealing. What is the process of generating DocBook XML from DUTS/DULG?

What would you need DocBook XML for?

Quoted 8 lines

It's true, I may have. :) On the other hand, it seems that there are still a lot of people who are in the same boat. Most manuals I have seen from other embedded companies (Freescale, Analog Devices, etc.) seem to provide their own format/content. Additionally, most companies will prefer to have their content formatted to match the rest of their manuals, which is easily done from DocBook XML. Again, if you can do that with DUTS/DULG, then that's great and probably eliminates the need for this tool.

Ah, I see.

Well, we use htmldoc to generate .pdf and .ps files from HTML; you could for example use Tidy to convert HTML to DocBook (and I'm sure there are lots of other tools that can do that).

Quoted 5 lines

Quoted 2 lines

Hm... we cannot look at your patches, you just posted the patch statistics, but no content.

This is just the patch series introduction, the actual content appears in 1-3.

Yes, I know. Sorry, patches arrived shortly after I started replying, and I didn;t notice.

Quoted 3 lines

As I mentioned, I'm not all that familiar with the abilities of DUTS/DULG, but I got the impression that there was still a fair amount of manual labor involved for each manual. Thus most people (that I'm

Indeed there is a lot of manual labor involved in preparing and updating the documentation (the framework, that is; once a board has been configured and added to DUTS, generating the needed include files is mostly an automatic procedure - just the upload of the files to the web server has to be triggered manually, but even this is scripted).

But you have to spend at least the same amount of writing and updating the documentation, don't you?

Quoted 3 lines

I was hoping that creating documentation from comments in the source would be easier to maintain and providing DocBook output would allow others to more easily reuse the U-Boot manual contents.

A very, very old version of the DULG was set up like this, using a set of Perl scripts and generating DokBoot output. We switched to the wiki based approach some 6 years ago, and I never had the slightest wish to switch back.

Add support to a small subset of commands, environment variables, and POSTs. These are meant to show the syntax and capabilities of the uboot-doc parser.

Signed-off-by John Schmoller <[email protected]>

---
 common/cmd_i2c.c    |   71 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 common/cmd_mem.c    |   45 +++++++++++++++++++++++++++++++-
 common/cmd_net.c    |   30 ++++++++++++++++++++-
 common/env_common.c |   35 +++++++++++++++++++++++++
 post/tests.c        |   16 +++++++++++
 5 files changed, 192 insertions(+), 5 deletions(-)

common/cmd_i2c.c +54 -1

index 8f0fc9e..14d394c 100644
@@ -1297,7 +1297,70 @@ int do_i2c(cmd_tbl_t * cmdtp, int flag, int argc, char *argv[])
 }
 
 /***************************************************/
-
+/**
+ * @cmd: i2c speed
+ * @sect: I2C
+ * @param[or]: speed - target I2C bus speed
+ * @desc: Show or set I2C bus speed
+ */
+/**
+ * @cmd: i2c dev
+ * @sect: I2C
+ * @param[or]: dev - I2C device to make current
+ * @desc: Show or set current I2C bus
+ */
+/**
+ * @cmd: i2c md
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param[or]: # of objects - Numer of bytes to read
+ * @desc: Read from I2C device
+ */
+/**
+ * @cmd: i2c mm
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @desc: Write to I2C device (auto-incrementing)
+ */
+/**
+ * @cmd: i2c mw
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param: value - Value to write to I2C address
+ * @param[or]: count - Number of bytes to write
+ * @desc: Write to I2C device (fill)
+ */
+/**
+ * @cmd: i2c nm
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @desc: Write to I2C device (constant address)
+ */
+/**
+ * @cmd: i2c crc32
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param: count - Number of bytes to CRC
+ * @desc: Computes the CRC32 checksum of an address range
+ */
+/**
+ * @cmd: i2c probe
+ * @sect: I2C

Dear John Schmoller,

Quoted 76 lines · In message <310e8398479f23a6bfaceccf6cc86631b7bff4dc.1248798202.git.jschmoller@…

Add support to a small subset of commands, environment variables, and POSTs. These are meant to show the syntax and capabilities of the uboot-doc parser.

Signed-off-by John Schmoller <[email protected]>

---
 common/cmd_i2c.c    |   71 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 common/cmd_mem.c    |   45 +++++++++++++++++++++++++++++++-
 common/cmd_net.c    |   30 ++++++++++++++++++++-
 common/env_common.c |   35 +++++++++++++++++++++++++
 post/tests.c        |   16 +++++++++++
 5 files changed, 192 insertions(+), 5 deletions(-)

common/cmd_i2c.c +54 -1

index 8f0fc9e..14d394c 100644
@@ -1297,7 +1297,70 @@ int do_i2c(cmd_tbl_t * cmdtp, int flag, int argc, char *argv[])
 }
 
 /***************************************************/
-
+/**
+ * @cmd: i2c speed
+ * @sect: I2C
+ * @param[or]: speed - target I2C bus speed
+ * @desc: Show or set I2C bus speed
+ */
+/**
+ * @cmd: i2c dev
+ * @sect: I2C
+ * @param[or]: dev - I2C device to make current
+ * @desc: Show or set current I2C bus
+ */
+/**
+ * @cmd: i2c md
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param[or]: # of objects - Numer of bytes to read
+ * @desc: Read from I2C device
+ */
+/**
+ * @cmd: i2c mm
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @desc: Write to I2C device (auto-incrementing)
+ */
+/**
+ * @cmd: i2c mw
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param: value - Value to write to I2C address
+ * @param[or]: count - Number of bytes to write
+ * @desc: Write to I2C device (fill)
+ */
+/**
+ * @cmd: i2c nm
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @desc: Write to I2C device (constant address)
+ */
+/**
+ * @cmd: i2c crc32
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param: count - Number of bytes to CRC
+ * @desc: Computes the CRC32 checksum of an address range
+ */
+/**
+ * @cmd: i2c probe
+ * @sect: I2C

common/cmd_i2c.c int do_i2c(cmd_tbl_t * cmdtp, int flag, int argc, char *argv[]) · line 1363

Quoted 10 lines

common/cmd_i2c.c +10 -0

+ * @desc: Shows all the devices on the I2C bus
+ */
+/**
+ * @cmd: i2c loop
+ * @sect: I2C
+ * @param: chip - I2C address of desired device
+ * @param: address[.0, .1, .2] - Address into I2C device
+ * @param: # of objects - Number of bytes to loop over
+ * @desc: Looping read of device.  Never returns.
+ */

Hm... isn't this basicly duplicating the information you get when running the "help" command? Then why don't you simply include that output here, like we do in the DULG?

To me this seems to be a mix of API documentation (which would be useful, but you do it for the wrong set of functions) and User's Manual (where it mostly seems to be a duplication of information, and maintaining redundant information has never been a good idea).

It's true, I was a little lax on commenting the I2C commands. If I were to move forward with this documentation tool, I would comment these commands more thoroughly like I did with the Memory commands.