Skip to contentSkip to conversation

Possible changes to the project docs

DISCUSSION Richard Purdie 7 participants 21 messages Apr 15, 2020 – Jun 15, 2020

lore
Share & tools
lore
#0 lore raw

Since Robert was talking about future docs wide changes, I wanted to put an idea out there. Its something a few of us have talked a little about. At this point it is an idea, not a decision and something we're thinking we need to explore and evaluate.

Basically, the thought is whether to move from docbook to sphinx.

Other projects like the kernel have switched and it could assist some of our website/publishing challenges. It might also be easier for people to contribute to and there are wider and more modern tools supporting that format.

It'd be a big change, particularly if we consider we may need to convert previous releases of the docs as well as the current version. That said, if we were going to do this, the time could be now...

Cheers,

Richard

#1 lore raw
Quoted 1 line · Richard Purdie

Basically, the thought is whether to move from docbook to sphinx.

This sounds like a good idea to me.

Prior to yesterday I had never looked into sphinx before. Skimming some information about sphinx, it looks like it's flexible with regards to the exact formatting you use for, for example, section headings. If that's true, I believe we'll need a "coding standard" in place before proceeding. We wouldn't want the BSP Manual to use:

	section 1
	^^^^^^^^^
	section 1.1
	===========
	section 1.1.1
	-------------

and the Reference Manual to use:

	section 1
	=========
	section 1.1
	***********
	section 1.1.1
	^^^^^^^^^^^^^

Best regards,
Trevor

#2 lore raw
Quoted 18 lines · Trevor Woerner
Quoted 1 line · Richard Purdie

Basically, the thought is whether to move from docbook to sphinx.

This sounds like a good idea to me.

Prior to yesterday I had never looked into sphinx before. Skimming some information about sphinx, it looks like it's flexible with regards to the exact formatting you use for, for example, section headings. If that's true, I believe we'll need a "coding standard" in place before proceeding. We wouldn't want the BSP Manual to use:

	section 1
	^^^^^^^^^
	section 1.1
	===========
	section 1.1.1
	-------------
Quoted 8 lines

and the Reference Manual to use:

	section 1
	=========
	section 1.1
	***********
	section 1.1.1
	^^^^^^^^^^^^^

absolutely, and i'll throw out one more "coding standard" that i
initially loathed, but came to appreciate -- keep sentences on the same physical line.

i first ran into that standard when contributing to the "pro git"
book, and i hated, hated, hated it as it naturally made long sentences go on forever off the right side of the terminal window.

however, i grew to like it because it makes side-by-side comparison
trivial and if you ended up with a sentence like that, it made you reconsider whether you were doing this correctly.

anyway, yes to the above.

rday

p.s. i've played briefly with sphinx on my fedora 31/32 system -- has anyone set up a full processing toolchain that can just be purloined to get started? even though it looks pretty simple.

#3 lore raw
Quoted 39 lines · Robert P. J. Day
Quoted 8 lines · Trevor Woerner
Quoted 1 line · Richard Purdie

Basically, the thought is whether to move from docbook to sphinx.

This sounds like a good idea to me.

Prior to yesterday I had never looked into sphinx before. Skimming some information about sphinx, it looks like it's flexible with regards to the exact formatting you use for, for example, section headings. If that's

true, I

Quoted 1 line

believe we'll need a "coding standard" in place before proceeding. We

wouldn't

Quoted 9 lines

want the BSP Manual to use:

      section 1
      ^^^^^^^^^
      section 1.1
      ===========
      section 1.1.1
      -------------
Quoted 8 lines

and the Reference Manual to use:

      section 1
      =========
      section 1.1
      ***********
      section 1.1.1
      ^^^^^^^^^^^^^

absolutely, and i'll throw out one more "coding standard" that i
initially loathed, but came to appreciate -- keep sentences on the same physical line.

i first ran into that standard when contributing to the "pro git"
book, and i hated, hated, hated it as it naturally made long sentences go on forever off the right side of the terminal window.

Quoted 6 lines

however, i grew to like it because it makes side-by-side comparison
trivial and if you ended up with a sentence like that, it made you reconsider whether you were doing this correctly.

anyway, yes to the above.

I added this in one of my commits messages, to start with:

==
sphinx: implement headings

The YP manuals layout is organized as:

Book
Chapter
Section
Section
Section

During the conversion with pandoc, some of the headings are not converted properly. Let's decide for now to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this change, we now have the same TOC with Sphinx and DocBook.
==

Quoted 6 lines

rday

p.s. i've played briefly with sphinx on my fedora 31/32 system -- has anyone set up a full processing toolchain that can just be purloined to get started? even though it looks pretty simple.

As I put in one of my emails before you can use my dev branch:

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

then it's just
pip3 install sphinx

# to use the non default theme
pip3 install sphinx_rtd_theme

cd documentation
make html

and it should generate exactly that on your local folder
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I am happy to take patches in this branch (or create/maintain a contrib branch if more folks want to play/experiment and send patches).

Quoted 1 line
#4 lore raw
Quoted 9 lines · Nicolas Dechesne

As I put in one of my emails before you can use my dev branch:

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

then it's just
pip3 install sphinx

# to use the non default theme
pip3 install sphinx_rtd_theme
Quoted 8 lines

cd documentation make html

and it should generate exactly that on your local folder
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I am happy to take patches in this branch (or create/maintain a contrib branch if more folks want to play/experiment and send patches).

definitely need some mechanism to handle docbook entities and basic
tags like <filename> ... i'm still poring over pandoc, can it be configured to deal with that?

will go read more ...

rday

#5 lore raw
Quoted 20 lines · Robert P. J. Day
Quoted 9 lines · Nicolas Dechesne

As I put in one of my emails before you can use my dev branch:

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

then it's just
pip3 install sphinx

# to use the non default theme
pip3 install sphinx_rtd_theme
Quoted 8 lines

cd documentation make html

and it should generate exactly that on your local folder
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I am happy to take patches in this branch (or create/maintain a contrib branch if more folks want to play/experiment and send patches).

Quoted 6 lines

definitely need some mechanism to handle docbook entities and basic
tags like <filename> ... i'm still poring over pandoc, can it be configured to deal with that?

will go read more ...

It does 'most of the time'. See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.html#bsp-layers

where the docbook source is:
Each repository is a BSP layer supported by the Yocto Project

(e.g. <filename>meta-raspberrypi</filename> and

<filename>meta-intel</filename>).

I am not sure if it always does the *right* thing though.. and I have seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex for 15% and then manually finish the work.. it might be a bit optimistic ;)

rday

Quoted 2 lines
#6 lore raw
Quoted 42 lines · Nicolas Dechesne

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <[email protected]> wrote:
On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev branch:
>
> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>
> then it's just
> pip3 install sphinx
> # to use the non default theme
> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or create/maintain a > contrib branch if more folks want to play/experiment and send > patches).

  definitely need some mechanism to handle docbook entities and basic tags like <filename> ... i'm still poring over pandoc, can it be configured to deal with that?

  will go read more ...

It does 'most of the time'. See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.html#bsp-layers

where the docbook source is:
       Each repository is a BSP layer supported by the Yocto Project

        (e.g. <filename>meta-raspberrypi</filename> and


        <filename>meta-intel</filename>).

Quoted 8 lines


 I am not sure if it always does the *right* thing though.. and I have seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex for 15% and then manually finish the work.. it might be a bit optimistic ;)

interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can still see the tag. i suspect there will be plenty of tweaking, but i'm still totally on-board with this, and am willing to contribute.

rday

#7 lore raw

Hi:

I'd like to actually start planning this conversion to Sphinx. So far myself, Rob D., Nicolas, and Tim O. are all interested in helping out.

I'd like to have a Zoom meeting or something similar to get started on where to go from Nico's proof of concept. Please let me know if you want to attend - I'm thinking sometime early next week.

Thanks,
-Mark M.

Quoted 72 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Robert P. J. Day
Sent: Thursday, April 16, 2020 11:06 AM
To: [email protected]
Subject: Re: [docs] Possible changes to the project docs

Quoted 3 lines · Nicolas Dechesne

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <[email protected]>

Quoted 41 lines · wrote:

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev branch:
>
> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>
> then it's just
> pip3 install sphinx
> # to use the non default theme
> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or create/maintain a > contrib branch if more folks want to play/experiment and send > patches).

  definitely need some mechanism to handle docbook entities and basic tags like <filename> ... i'm still poring over pandoc, can it be configured to deal with that?

  will go read more ...

It does 'most of the time'. See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.
html#bsp-layers

where the docbook source is:
       Each repository is a BSP layer supported by the Yocto Project

        (e.g. <filename>meta-raspberrypi</filename> and

        <filename>meta-intel</filename>).

Quoted 7 lines

 I am not sure if it always does the *right* thing though.. and I have seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex for 15% and then manually finish the work.. it might be a bit optimistic ;)

interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can still see the tag. i suspect there will be plenty of tweaking, but i'm still totally on-board with this, and am willing to contribute.

rday

#8 lore raw

hi Mark,

Quoted 9 lines · Mark Morton

Hi:

I'd like to actually start planning this conversion to Sphinx. So far myself, Rob D., Nicolas, and Tim O. are all interested in helping out.

I'd like to have a Zoom meeting or something similar to get started on where to go from Nico's proof of concept. Please let me know if you want to attend - I'm thinking sometime early next week.

Thanks for starting this. I will be happy to help. I can prepare an 'initial' plan to get us started.

Quoted 84 lines

Thanks,
-Mark M.

Quoted 2 lines

-----Original Message-----
From: [email protected] <[email protected]> On

Behalf

Quoted 10 lines

Of Robert P. J. Day
Sent: Thursday, April 16, 2020 11:06 AM
To: [email protected]
Subject: Re: [docs] Possible changes to the project docs

Quoted 3 lines · Nicolas Dechesne

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <

[email protected]>

Quoted 6 lines
Quoted 5 lines · wrote:

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev branch:
>
>

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

Quoted 15 lines
Quoted 15 lines

>
> then it's just
> pip3 install sphinx
> # to use the non default theme
> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or create/maintain a > contrib branch if more folks want to play/experiment and send > patches).

Quoted 1 line
Quoted 1 line

definitely need some mechanism to handle docbook entities and

basic

Quoted 28 lines
Quoted 20 lines

tags like <filename> ... i'm still poring over pandoc, can it be configured to deal with that?

will go read more ...

It does 'most of the time'. See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.
html#bsp-layers

where the docbook source is:
Each repository is a BSP layer supported by the Yocto Project

(e.g. <filename>meta-raspberrypi</filename> and

<filename>meta-intel</filename>).

Quoted 7 lines

I am not sure if it always does the *right* thing though.. and I have
seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex for 15% and then manually finish the work.. it might be a bit optimistic ;)

Quoted 6 lines

interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can

still see

Quoted 1 line

the tag. i suspect there will be plenty of tweaking, but i'm still

totally on-board

Quoted 3 lines

with this, and am willing to contribute.

rday

#9 lore raw
Quoted 11 lines · Nicolas Dechesne

hi Mark,

Quoted 5 lines · Mark Morton

Hi:

I'd like to actually start planning this conversion to Sphinx. So far myself, Rob D., Nicolas, and Tim O. are all interested in helping out.

Quoted 8 lines
Quoted 4 lines

I'd like to have a Zoom meeting or something similar to get started on where to go from Nico's proof of concept. Please let me know if you want to attend - I'm thinking sometime early next week.

Thanks for starting this. I will be happy to help. I can prepare an 'initial' plan to get us started.

Well.. I've started this wiki:
https://wiki.yoctoproject.org/wiki/Documentation_Sphinx

Quoted 89 lines
Quoted 88 lines

Thanks,
-Mark M.

Quoted 2 lines

-----Original Message-----
From: [email protected] <[email protected]> On

Behalf

Quoted 10 lines

Of Robert P. J. Day
Sent: Thursday, April 16, 2020 11:06 AM
To: [email protected]
Subject: Re: [docs] Possible changes to the project docs

Quoted 3 lines · Nicolas Dechesne

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <

[email protected]>

Quoted 6 lines
Quoted 5 lines · wrote:

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev branch:
>
>

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

Quoted 4 lines
Quoted 4 lines

>
> then it's just
> pip3 install sphinx
> # to use the non default theme

Quoted 8 lines
Quoted 8 lines

> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or create/maintain

a

Quoted 4 lines
Quoted 4 lines

> contrib branch if more folks want to play/experiment and send > patches).

definitely need some mechanism to handle docbook entities and

basic

Quoted 8 lines
Quoted 8 lines

tags like <filename> ... i'm still poring over pandoc, can it be configured to deal with that?

will go read more ...

It does 'most of the time'. See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml

.

Quoted 20 lines
Quoted 12 lines

html#bsp-layers

where the docbook source is:
Each repository is a BSP layer supported by the Yocto Project

(e.g. <filename>meta-raspberrypi</filename> and

<filename>meta-intel</filename>).

Quoted 7 lines

I am not sure if it always does the *right* thing though.. and I have
seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex for 15% and then manually finish the work.. it might be a bit optimistic ;)

Quoted 6 lines

interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can

still see

Quoted 1 line

the tag. i suspect there will be plenty of tweaking, but i'm still

totally on-board

Quoted 3 lines

with this, and am willing to contribute.

rday

#10 lore raw

Hmm, it seems the interwebs ate my prior message.   TLDR;

Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora

* A switch from DocBook is a net benefit regardless of what you switch too.
* Making docs more accessible to dev and user contributors is the key.
* I regard ascidoc/md/rst as near enough that they don't swing the decision - in terms of being approachable to devs and users.
* A potential discriminator is l10n/i18n - on the face of it they all seem to be imperfect (wip) and adequate.
* Is there any other strategic benefit to be obtained?

I think so.  Looking at https://antora.org/ this stands out:
*The ability to retrieve and aggregate all the content from different git repositories.*

This provides the following headroom:
* Move documentation into the repo that holds the related code.  Bringing documentation closer to devs who are more likely to write some preliminary documentation and correct documentation errors/ambiguities.
* Allow individual repo's/tools to evolve at their own pace - a release that repo/tool X would like to make 'yesterday' can be made and everyone can be confident the docs are still consistent (assumes that tool/repo-Y does not make any statements about tool/repo-X in the Y-docs)
* Allow the project to treat a repo as a 1st class citizen even if it is not under the openembedded.org or yoctoproject.org
* Corollary: Allow any repo outside of openembedded.org or yoctoproject.org to incorporate whichever of the tools/repos they need (git cloning & forking model allows/encourages that) and present documentation that is integrated and targeted (Antora could - eventually - allow/encourage that).
* Give the Yocto-project flexibility in how to organize itself going forward - documentation is not an impediment.

Of course there is a learning curve and the docs need to be refactored back to their home tool/repo. To my mind there is a categorical distinction that can be made between the DocBook alternatives.

The questions is whether any of the above conjectures are of value or attractive?

HTH?
--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

#11 lore raw

hi Mark,

thanks a lot for joining the discussion! This is very helpful and much appreciated.

Quoted 2 lines · Mark Van De Vyver

Hmm, it seems the interwebs ate my prior message. TLDR;

I believe you replied privately to me, your message is not lost and is in good hands , well mine ;)

Quoted 7 lines

Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora

* A switch from DocBook is a net benefit regardless of what you switch too.
* Making docs more accessible to dev and user contributors is the key.
* I regard ascidoc/md/rst as near enough that they don't swing the decision - in terms of being approachable to devs and users.

* A potential discriminator is l10n/i18n - on the face of it they all seem

Quoted 2 lines

to be imperfect (wip) and adequate.

We don't have plans for that now, even if it might be a good thing for the project, I think we struggle enough with one language today.

Quoted 2 lines

* Is there any other strategic benefit to be obtained?

The main goal is to ensure documentation is 'easier' to read and write, encouraging more contributors, while keeping the reputation of the quality of the project's documentation.

Quoted 4 lines

I think so. Looking at https://antora.org/ this stands out:
*The ability to retrieve and aggregate all the content from different git repositories.*

I think there are 2 sides to the problem:
1. the format of the documentation: DocBook (XML) vs reST vs Asciidoc. I think it's well accepted that Markdown is 'not enough' for large tech documentation/book
2. the tools available to process the documentation (sphinx, asciidoctor, antora.. ). Antora is a toolset on top of Asciidoc.

The main reasons why we've started to explore/experiment with reST/Sphinx (as opposed to Asciidoc) are:
1. this is what Python uses, and our project is intimately linked to Python. Sphinx can be extended with plugins in Python, and we have very capable Python developers around here.
2. we've closely studied what happened in the Linux kernel community, and their thought process when they moved away from Docbook a couple of years ago, which is nicely summarized in the following articles:
https://lwn.net/Articles/671496/
https://lwn.net/Articles/692704/
https://lwn.net/Articles/692705/
Even though the initial plan was to move from Docbook to Asciidoc, they ended up using reST/Sphinx in the end. Given the proximity of the YP and kernel communities, we should pay close attention to what they've done.

That said, no decision has been made, Sphinx is just the first proof of concept which have been made. So now is definitely the right time to discuss all options!

Quoted 6 lines

This provides the following headroom:
* Move documentation into the repo that holds the related code. Bringing documentation closer to devs who are more likely to write some preliminary documentation and correct documentation errors/ambiguities.

I agree that the project documentation would benefit from some inline documentation. Not for everything, but maybe for classes, variables, functions, it would be easier to maintain if the documentation was 'closer' to the doc. The kernel documentation uses a mix of inline doc vs separate doc already (with sphinx).

Quoted 6 lines

* Allow individual repo's/tools to evolve at their own pace - a release that repo/tool X would like to make 'yesterday' can be made and everyone can be confident the docs are still consistent (assumes that tool/repo-Y does not make any statements about tool/repo-X in the Y-docs)
* Allow the project to treat a repo as a 1st class citizen even if it is not under the openembedded.org or yoctoproject.org

I don't think there is such a concept of a 1st class citizen. The Openembedded ecosystem is made of a very large amount of layers, and only a minority of layers are hosted on oe.org or yp.org. I believe that all layers are treated equally, as soon as they are listed/added to the OE layer index maybe.

Quoted 6 lines

* Corollary: Allow any repo outside of openembedded.org or yoctoproject.org to incorporate whichever of the tools/repos they need (git cloning & forking model allows/encourages that) and present documentation that is integrated and targeted (Antora could - eventually - allow/encourage that).

The YP documentation 'project' is about publishing documentation for what makes up the Yocto Project (OE-core, bitbake, ..). The current Docbook structure is probably too complex (learning curve) to be adopted in other layers. So anything we could improve to make it simpler for other layers to benefit from doc infrastructure would be a nice bonus.

Quoted 8 lines

* Give the Yocto-project flexibility in how to organize itself going forward - documentation is not an impediment.

Of course there is a learning curve and the docs need to be refactored back to their home tool/repo.
To my mind there is a categorical distinction that can be made between the DocBook alternatives.

Can you provide more information about what Antora vs Asciidoc/asciidoctor?

Quoted 12 lines

The questions is whether any of the above conjectures are of value or attractive?

HTH?
--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

#12 lore raw

On Thu, 28 May 2020, Nicolas Dechesne wrote:

... snip ...

Quoted 6 lines

The main reasons why we've started to explore/experiment with reST/Sphinx (as opposed to Asciidoc) are:
1. this is what Python uses, and our project is intimately linked to Python. Sphinx can be extended with plugins in Python, and we have very capable Python developers around here.
2. we've closely studied what happened in the Linux kernel community, and their thought

Quoted 8 lines

process when they moved away from Docbook a couple of years ago, which is nicely summarized in the following articles:
https://lwn.net/Articles/671496/
https://lwn.net/Articles/692704/
https://lwn.net/Articles/692705/
Even though the initial plan was to move from Docbook to Asciidoc, they ended up using reST/Sphinx in the end. Given the proximity of the YP and kernel communities, we should pay close attention to what they've done.

... snip ...

FWIW, the red hat folks working on podman/libpod docs are moving to
sphinx as well.

rday

#13 lore raw
In reply to Nicolas Dechesne
Quoted 3 lines · Nicolas Dechesne

hi Mark,

thanks a lot for joining the discussion! This is very helpful and much appreciated.

... snip ...

Quoted 3 lines

Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora

* A switch from DocBook is a net benefit regardless of what you switch too.

the one aspect of docbook i will miss is actual semantic markup
which can then be customized with stylesheets, but that benefit is outweighed by all the downsides of the complexity. and given that i'm poring over the sphinx docs right now and playing with it, yeah, sphinx is my choice. :-)

rday

#14 lore raw
Quoted 12 lines · Robert P. J. Day
Quoted 3 lines · Nicolas Dechesne

hi Mark,

thanks a lot for joining the discussion! This is very helpful and much

appreciated.

... snip ...

Quoted 1 line

Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and

Antora

Quoted 1 line
Quoted 9 lines
Quoted 1 line

* A switch from DocBook is a net benefit regardless of what you

switch too.

the one aspect of docbook i will miss is actual semantic markup
which can then be customized with stylesheets, but that benefit is outweighed by all the downsides of the complexity. and given that i'm poring over the sphinx docs right now and playing with it, yeah, sphinx is my choice. :-)

Ah, nice. If you have any patches that are relevant for the sphinx proof of concept, please share them. I started a sphinx2 branch here:
https://git.linaro.org/people/nicolas.dechesne/yocto-docs.git/log/?h=sphinx2

with a 'better' commit log structure. I have also done the pandoc conversion for all manuals.

Quoted 3 lines

rday

#15 lore raw
In reply to Nicolas Dechesne
Quoted 25 lines · Nicolas Dechesne

hi Mark,

thanks a lot for joining the discussion! This is very helpful and much appreciated.

Quoted 1 line · Mark Van De Vyver

Hmm, it seems the interwebs ate my prior message.   TLDR;

I believe you replied privately to me, your message is not lost and is in good hands , well mine ;)

Quoted 7 lines

Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora

* A switch from DocBook is a net benefit regardless of what you switch too.
* Making docs more accessible to dev and user contributors is the key.
* I regard ascidoc/md/rst as near enough that they don't swing the decision - in terms of being approachable to devs and users.

Quoted 8 lines
Quoted 2 lines

* A potential discriminator is l10n/i18n - on the face of it they all seem to be imperfect (wip) and adequate.

We don't have plans for that now, even if it might be a good thing for the project, I think we struggle enough with one language today.

Agreed.
Unless there is a group dedicated to a translation, it would only make sense to restrict the translation to some automated service - google translate at worse or some thing like Transifex. Transifex offer a freelicense to "Transifex for free to Open Source projects ( http://docs.transifex.com/faq/#what-is-considered-an-open-source-project ) that have no funding, revenue, and/or commercialization model".  Not clear if we qualify for that. The question is whether the cost of a 'rough' translation is worth the benefit of new users who are technically profficient but whose English is not at a level to grok the docs fully.
Thinking here of the proverbial 14yo who has just got board/device xyz and stumbles upon yoctoproject.org.

Quoted 15 lines
Quoted 4 lines

* Is there any other strategic benefit to be obtained?

The main goal is to ensure documentation is 'easier' to read and write, encouraging more contributors, while keeping the reputation of the quality of the project's documentation.

Quoted 3 lines

I think so.  Looking at https://antora.org/ this stands out:
*The ability to retrieve and aggregate all the content from different git repositories.*

Quoted 6 lines

I think there are 2 sides to the problem:
1. the format of the documentation: DocBook (XML) vs reST vs Asciidoc. I think it's well accepted that Markdown is 'not enough' for large tech documentation/book

MD is ruled out, to my mind, by the lack of multi-repo support;
However it is worth noting the scope of the syntax/functionality should be sufficient - is see here ( https://cloudflow.io/docs/current ) for a larger project and here ( https://jlenv.github.io/jlenv ) for my Julia port of rbenv (devs here might be familiar with the Python port of rbenv). Hence, I would argue that, which ever of asciidoc or rst is used, we encourage users employ to a subset of the syntax to keep the docs simple and maintainable.
AFAICT we're not dealing Julia or Linux graphics code levels of complexity.  Happy to be corrected.

Quoted 11 lines

2. the tools available to process the documentation (sphinx, asciidoctor, antora.. ). Antora is a toolset on top of Asciidoc.

The main reasons why we've started to explore/experiment with reST/Sphinx (as opposed to Asciidoc) are:
1. this is what Python uses, and our project is intimately linked to Python. Sphinx can be extended with plugins in Python, and we have very capable Python developers around here.

Developer productivity is key.
I'd argue constraining to tool X's functionality.
Again, I'm happy to be wrong, yet AFAICT there shouldn't be a level of complexity in this projects documentation requirements that exceeds these tools.

Quoted 12 lines

2. we've closely studied what happened in the Linux kernel community, and their thought process when they moved away from Docbook a couple of years ago, which is nicely summarized in the following articles:
https://lwn.net/Articles/671496/
https://lwn.net/Articles/692704/
https://lwn.net/Articles/692705/
Even though the initial plan was to move from Docbook to Asciidoc, they ended up using reST/Sphinx in the end. Given the proximity of the YP and kernel communities, we should pay close attention to what they've done.

Agreed - I read that transition slightly differently: The kernel doc tooling was imperfect but generally tenable.  The Linux graphics code base complexity made the docs untenable, and forced an alternative to be sought. Once the graphics code base needs were satisfied it obviously made sense to standardize.  Just as Julia has use cases of that compexity. Are we really dealing with that level of difficulty (as disctinct from very elborate/long chains of scripts/routines - each conceptually relatively straightword in isloation).

Quoted 7 lines

That said, no decision has been made, Sphinx is just the first proof of concept which have been made. So now is definitely the right time to discuss all options!

Great. So I spent some time digging further (See Transifex note above)...

Quoted 7 lines
Quoted 6 lines

This provides the following headroom:
* Move documentation into the repo that holds the related code.  Bringing documentation closer to devs who are more likely to write some preliminary documentation and correct documentation errors/ambiguities.

Quoted 6 lines

I agree that the project documentation would benefit from some inline documentation. Not for everything, but maybe for classes, variables, functions, it would be easier to maintain if the documentation was 'closer' to the doc. The kernel documentation uses a mix of inline doc vs separate doc already (with sphinx).

Agreed. Pretty much every tool can handle some level of rendering inline docs that should be adequate. What Antora allows is doc sharing between repos, and AFAICT, Sphinx doesn't provide that headroom.

Quoted 12 lines
Quoted 11 lines

* Allow individual repo's/tools to evolve at their own pace - a release that repo/tool X would like to make 'yesterday' can be made and everyone can be confident the docs are still consistent (assumes that tool/repo-Y does not make any statements about tool/repo-X in the Y-docs)
* Allow the project to treat a repo as a 1st class citizen even if it is not under the openembedded.org ( http://openembedded.org ) or yoctoproject.org ( http://yoctoproject.org )

Quoted 7 lines

I don't think there is such a concept of a 1st class citizen. The Openembedded ecosystem is made of a very large amount of layers, and only a minority of layers are hosted on oe.org ( http://oe.org ) or yp.org (
http://yp.org ). I believe that all layers are treated equally, as soon as they are listed/added to the OE layer index maybe.

Yet such layers cannot present integrated documentation. Lets take a concrete existing example:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-amd/tree/?h=dunfell

Ideally there would be a way for that project to present their (currently) secondary documentation (assume the eight (8) md files are ported to which ever syntax or rst or asciidoc) integrated with the yocto/oe parts of their documentation.  Even if this is only for some group internal document portal or readthedocs or some such portal. In fact on the dunfell branch the current DOCUMENTATION.md suggests a Antora " playbook ( https://docs.antora.org/antora/2.2/playbook/ ) " (untested):

site:
title: Layer XYZ Documentation

content:
sources:
- url: .
branches: dunfell
start_path: documentation/introduction - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/overview-manual - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ brief-yoctoprojectqs ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
- url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ toaster-manual ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
- url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ sdk-manual ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
output: clean: true dir: public
Take a second concrete example:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi

The proverbial 12-14yo with a board now has three sets of documentation to contend with (following the README.md):
1. https://meta-raspberrypi.readthedocs.io
2. Yocto/OE
3. raspberrypi.org

Quoted 10 lines
Quoted 8 lines

* Corollary: Allow any repo outside of openembedded.org (
http://openembedded.org ) or yoctoproject.org ( http://yoctoproject.org ) to incorporate whichever of the tools/repos they need (git cloning & forking model allows/encourages that) and present documentation that is integrated and targeted (Antora could - eventually - allow/encourage that).

Quoted 3 lines

The YP documentation 'project' is about publishing documentation for what makes up the Yocto Project (OE-core, bitbake, ..).

Agreed. I suppose I'm arguing the project specific elements of the documentation (i.e not reusable by/of interest to a repo/layer) could (ideally) be limited to page header, footer and brand image detail - everything else could be reusable by layers as they think best fits their userbase/audience.

Quoted 10 lines

The current Docbook structure is probably too complex (learning curve) to be adopted in other layers. So anything we could improve to make it simpler for other layers to benefit from doc infrastructure would be a nice bonus.

Quoted 3 lines

* Give the Yocto-project flexibility in how to organize itself going forward - documentation is not an impediment.

Quoted 10 lines
Quoted 4 lines

Of course there is a learning curve and the docs need to be refactored back to their home tool/repo.
To my mind there is a categorical distinction that can be made between the DocBook alternatives.

Can you provide more information about what Antora vs Asciidoc/asciidoctor?

Personally I've never encountered the need use asciidoc. Until now - I'll likely refactor the github.com/jlenv org documents so that each repo's documentation is pulled into a whole under jlenv.github.io/. My understanding Antora builds on asciidoctor.js.
I'd argue for whichever tool is used: Project and layer docs should limit themselves to that tools built-in functionality - patching a doc tool detracts dev cycles.
I'd argue to limit documentation to what is effectively the markdown level of functionality - but layer/repo maintainers are best placed to make this call.

Kind regards
Mark

Quoted 6 lines
Quoted 6 lines

The questions is whether any of the above conjectures are of value or attractive?

Quoted 9 lines
Quoted 7 lines

HTH?
--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

#16 lore raw
In reply to Nicolas Dechesne

hi Mark (Morton)

Quoted 21 lines · Nicolas Dechesne
Quoted 16 lines · Nicolas Dechesne

hi Mark,

Quoted 9 lines · Mark Morton

Hi:

I'd like to actually start planning this conversion to Sphinx. So far myself, Rob D., Nicolas, and Tim O. are all interested in helping out.

I'd like to have a Zoom meeting or something similar to get started on where to go from Nico's proof of concept. Please let me know if you want to attend - I'm thinking sometime early next week.

Quoted 7 lines
Quoted 3 lines

Thanks for starting this. I will be happy to help. I can prepare an 'initial' plan to get us started.

Well.. I've started this wiki:
https://wiki.yoctoproject.org/wiki/Documentation_Sphinx

Can we find a time next week to get things going? I am in CET time zone, fwiw. How about any of the following time (I don't know where you are so using PST timezone)

* Monday 9AM PST
* Tuesday 10AM PST
* Wed 9AM or 10AM PST

Quoted 95 lines
Quoted 95 lines
Quoted 94 lines

Thanks,
-Mark M.

Quoted 2 lines

-----Original Message-----
From: [email protected] <[email protected]> On

Behalf

Quoted 10 lines

Of Robert P. J. Day
Sent: Thursday, April 16, 2020 11:06 AM
To: [email protected]
Subject: Re: [docs] Possible changes to the project docs

Quoted 3 lines · Nicolas Dechesne

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <

[email protected]>

Quoted 4 lines
Quoted 3 lines · wrote:

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev

branch:

Quoted 2 lines
Quoted 2 lines
      >
      >

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

Quoted 4 lines
Quoted 4 lines

>
> then it's just
> pip3 install sphinx
> # to use the non default theme

Quoted 8 lines
Quoted 8 lines

> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or

create/maintain a

Quoted 4 lines
Quoted 4 lines

> contrib branch if more folks want to play/experiment and send > patches).

definitely need some mechanism to handle docbook entities

and basic

Quoted 1 line
Quoted 1 line

tags like <filename> ... i'm still poring over pandoc, can it

be

Quoted 7 lines
Quoted 7 lines

configured to deal with that?

will go read more ...

It does 'most of the time'. See:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.

Quoted 6 lines
Quoted 6 lines

html#bsp-layers

where the docbook source is:
Each repository is a BSP layer supported by the Yocto Project

Quoted 7 lines
Quoted 7 lines

(e.g. <filename>meta-raspberrypi</filename> and

<filename>meta-intel</filename>).

I am not sure if it always does the *right* thing though.. and I

have

Quoted 4 lines
Quoted 4 lines

seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex

for

Quoted 9 lines
Quoted 2 lines

15% and then manually finish the work.. it might be a bit optimistic ;)

interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can

still see

Quoted 1 line

the tag. i suspect there will be plenty of tweaking, but i'm still

totally on-board

Quoted 3 lines

with this, and am willing to contribute.

rday

#17 lore raw
Quoted 34 lines · Nicolas Dechesne

hi Mark (Morton)

Quoted 21 lines · Nicolas Dechesne
Quoted 16 lines · Nicolas Dechesne

hi Mark,

Quoted 9 lines · Mark Morton

Hi:

I'd like to actually start planning this conversion to Sphinx. So far myself, Rob D., Nicolas, and Tim O. are all interested in helping out.

I'd like to have a Zoom meeting or something similar to get started on where to go from Nico's proof of concept. Please let me know if you want to attend - I'm thinking sometime early next week.

Quoted 7 lines
Quoted 3 lines

Thanks for starting this. I will be happy to help. I can prepare an 'initial' plan to get us started.

Well.. I've started this wiki:
https://wiki.yoctoproject.org/wiki/Documentation_Sphinx

Quoted 8 lines

Can we find a time next week to get things going? I am in CET time zone, fwiw. How about any of the following time (I don't know where you are so using PST timezone)

* Monday 9AM PST
* Tuesday 10AM PST
* Wed 9AM or 10AM PST

Just to update everyone, we have agreed to have a first meeting this week, on Wed at 9AM PST time. The meeting has been added in the YP Google Calendar as well (refer to
https://www.yoctoproject.org/public-virtual-meetings/)

And I am sharing the meeting details here as well

https://zoom.us/j/97588912425
<https://www.google.com/url?q=https://zoom.us/j/97588912425&sa=D&ust=1592682564242000&usg=AOvVaw261x8lemP-eSIOdWUOiTEc>
Meeting ID: 975 8891 2425 One tap mobile +12532158782,,97588912425# US (Tacoma) +13462487799,,97588912425# US (Houston) Dial by your location +1 253 215 8782 US (Tacoma) +1 346 248 7799 US (Houston) +1 669 900 6833 US (San Jose) +1 301 715 8592 US (Germantown) +1 312 626 6799 US (Chicago) +1 646 558 8656 US (New York) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +44 208 080 6591 United Kingdom +44 208 080 6592 United Kingdom +44 330 088 5830 United Kingdom +44 131 460 1196 United Kingdom +44 203 481 5237 United Kingdom +44 203 481 5240 United Kingdom 0 800 260 5801 United Kingdom Toll-free 0 800 031 5717 United Kingdom Toll-free +60 3 9212 1727 Malaysia +60 3 3099 2229 Malaysia 1 800 818 425 Malaysia Toll-free +55 21 3958 7888 Brazil +55 11 4680 6788 Brazil 0 800 878 3108 Brazil Toll-free 0 800 282 5751 Brazil Toll-free +52 229 910 0061 Mexico +52 554 161 4288 Mexico +52 556 826 9800 Mexico 800 733 4118 Mexico Toll-free 800 062 7287 Mexico Toll-free +64 9 884 6780 New Zealand +64 4 886 0026 New Zealand 0 800 527 544 New Zealand Toll-free 0 800 002 260 New Zealand Toll-free Meeting ID: 975 8891 2425
Find your local number: https://zoom.us/u/adB5zrMbQ8
<https://www.google.com/url?q=https://zoom.us/u/adB5zrMbQ8&sa=D&ust=1592682564242000&usg=AOvVaw0DXi0h4wUmkZ03JQTC7xqf>

Quoted 97 lines
Quoted 97 lines
Quoted 97 lines
Quoted 96 lines

Thanks,
-Mark M.

Quoted 2 lines

-----Original Message-----
From: [email protected] <[email protected]> On

Behalf

Quoted 10 lines

Of Robert P. J. Day
Sent: Thursday, April 16, 2020 11:06 AM
To: [email protected]
Subject: Re: [docs] Possible changes to the project docs

Quoted 3 lines · Nicolas Dechesne

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <

[email protected]>

Quoted 4 lines
Quoted 3 lines · wrote:

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev

branch:

Quoted 2 lines
Quoted 2 lines
      >
      >

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

Quoted 4 lines
Quoted 4 lines

>
> then it's just
> pip3 install sphinx
> # to use the non default theme

Quoted 8 lines
Quoted 8 lines

> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or

create/maintain a

Quoted 1 line
Quoted 1 line

> contrib branch if more folks want to play/experiment and

send

Quoted 3 lines
Quoted 3 lines

> patches).

definitely need some mechanism to handle docbook entities

and basic

Quoted 1 line
Quoted 1 line

tags like <filename> ... i'm still poring over pandoc, can it

be

Quoted 7 lines
Quoted 7 lines

configured to deal with that?

will go read more ...

It does 'most of the time'. See:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.

Quoted 6 lines
Quoted 6 lines

html#bsp-layers

where the docbook source is:
Each repository is a BSP layer supported by the Yocto Project

Quoted 7 lines
Quoted 7 lines

(e.g. <filename>meta-raspberrypi</filename> and

<filename>meta-intel</filename>).

I am not sure if it always does the *right* thing though.. and I

have

Quoted 4 lines
Quoted 4 lines

seen occurrences where it failed. I am wondering if we should 'fix' pandoc , or just spend some time fixing the output of pandoc. I don't think we will avoid reading/checking each manual, anyways.. I was thinking we could use pandoc for 80% of the work, some regex

for

Quoted 9 lines
Quoted 2 lines

15% and then manually finish the work.. it might be a bit optimistic ;)

interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can

still see

Quoted 1 line

the tag. i suspect there will be plenty of tweaking, but i'm still

totally on-board

Quoted 3 lines

with this, and am willing to contribute.

rday

#18 lore raw
In reply to Richard Purdie

hi,

Quoted 7 lines · Richard Purdie

Since Robert was talking about future docs wide changes, I wanted to put an idea out there. Its something a few of us have talked a little about. At this point it is an idea, not a decision and something we're thinking we need to explore and evaluate.

Basically, the thought is whether to move from docbook to sphinx.

Good timing. I have been tossing around this idea... As much as I love using and reading the YP docs/manual, I know that maintaining and updating the documentation 'sources' is rather difficult.

Quoted 5 lines

Other projects like the kernel have switched and it could assist some of our website/publishing challenges. It might also be easier for people to contribute to and there are wider and more modern tools supporting that format.

Right, and they claimed in https://lwn.net/Articles/692704/:
"if the documentation can be written in plain text rather than DocBook XML, it's more likely to be written in the first place."

I have spent no more than a couple of hours on it, trying to understand why the kernel moved and how Sphinx works. The outcome is really interesting...

I use a conversion tool for DocBook .xml to Sphinx/ReST file format (pandoc). Of course there are conversion gothas.. but the documentation source code becomes much easier.

See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-release-process.xml.rst.txt
vs
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-release-process.xml

Or even for the variable glossary (sphinx has builtin support for that):
https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-variables.xml.rst

vs
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-variables.xml

I have picked the same theme as the Linux kernel documentation, but of course, the theme is just HTML and CSS, so it's entirely customizable.

For now, only 3 manuals were converted, the html output looks like that:
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I have made a couple of patches to show how to get there:
https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

It was straightforward to install anything i needed:
sudo apt install pandoc
pip3 install sphinx

# to use the non default theme
pip3 install sphinx_rtd_theme

then it's just:
make html

Quoted 4 lines

It'd be a big change, particularly if we consider we may need to convert previous releases of the docs as well as the current version. That said, if we were going to do this, the time could be now...

The intent is just to show where this could lead us. We need to ensure that it will not affect the quality of our documentation, which is obviously one of the project's gems!

Quoted 10 lines

Cheers,

Richard

#19 lore raw
Quoted 11 lines · Nicolas Dechesne

hi,

On Wed, Apr 15, 2020 at 12:19 PM Richard Purdie <[email protected] <mailto:[email protected]>> wrote:

Since Robert was talking about future docs wide changes, I wanted to put an idea out there. Its something a few of us have talked a little about. At this point it is an idea, not a decision and something we're thinking we need to explore and evaluate.

Quoted 6 lines

Basically, the thought is whether to move from docbook to sphinx.

Good timing. I have been tossing around this idea... As much as I love using and reading the YP docs/manual, I know that maintaining and updating the documentation 'sources' is rather difficult.

its not that bad IMHO : )

Quoted 12 lines

Other projects like the kernel have switched and it could assist some of our website/publishing challenges. It might also be easier for people to contribute to and there are wider and more modern tools supporting that format.

Right, and they claimed in https://lwn.net/Articles/692704/:
"if the documentation can be written in plain text rather than DocBook XML, it's more likely to be written in the first place."

Quoted 7 lines

I have spent no more than a couple of hours on it, trying to understand why the kernel moved and how Sphinx works. The outcome is really interesting...

I use a conversion tool for DocBook .xml to Sphinx/ReST file format (pandoc). Of course there are conversion gothas.. but the documentation source code becomes much easier.

Cool, a conversion tool. will have to look into it.

Quoted 30 lines

See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-release-process.xml.rst.txt
vs
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-release-process.xml

Or even for the variable glossary (sphinx has builtin support for that):
https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-variables.xml.rst
vs
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-variables.xml

I have picked the same theme as the Linux kernel documentation, but of course, the theme is just HTML and CSS, so it's entirely customizable.

For now, only 3 manuals were converted, the html output looks like that:
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I have made a couple of patches to show how to get there:
https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

It was straightforward to install anything i needed:
sudo apt install pandoc
pip3 install sphinx

# to use the non default theme
pip3 install sphinx_rtd_theme

then it's just:
make html

Quoted 8 lines

It'd be a big change, particularly if we consider we may need to convert previous releases of the docs as well as the current version. That said, if we were going to do this, the time could be now...

The intent is just to show where this could lead us. We need to ensure that it will not affect the quality of our documentation, which is obviously one of the project's gems!

I am in the middle of doing clean up work to remove "Common" code into their own files. This allow us to make a fix in one place instead of updating all the manuals.

This is what happens for every new dot release.  I did that for Zeus and Warrior and is silly to have to do and is what prompted to clean this up.  I hope sphinx will allow a similar structure as updating 11 files with the same information is not a good use of time.

- armin

Quoted 13 lines

Cheers,

Richard

#20 lore raw
In reply to Richard Purdie
Quoted 12 lines · Richard Purdie

Since Robert was talking about future docs wide changes, I wanted to put an idea out there. Its something a few of us have talked a little about. At this point it is an idea, not a decision and something we're thinking we need to explore and evaluate.

Basically, the thought is whether to move from docbook to sphinx.

Other projects like the kernel have switched and it could assist some of our website/publishing challenges. It might also be easier for people to contribute to and there are wider and more modern tools supporting that format.

Quoted 4 lines

It'd be a big change, particularly if we consider we may need to convert previous releases of the docs as well as the current version. That said, if we were going to do this, the time could be now...

ok, time to weigh in now that i have a minute ... i am totally on
board with a rewriting in sphinx, as that would not only make it easier for others to contribute, but it's something that, given some minimal direction, could be handed off to an intern of some kind or -- as i see already mentioned in subsequent replies -- fed through an automatic translator (eg., pandoc).

it would also be the right time to restructure some of the docs
(migration manual all on its own, etc, etc.).

the only downside to the switch is that, as arcane as docbook and
xsl stylesheets can be at times, they do support pretty much infinite and incredibly refined customization. however, i don't think the YP docs really scream for that level of customization.

i'm for it.

rday