Skip to contentSkip to conversation

Path troubles when using SPHINXDIRS option

DISCUSSION Mauro Carvalho Chehab 2 participants 8 messages Apr 21, 2020 – Apr 22, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad behavior, I tested (using the latest patch version) Sphinx from version 1.7.9 up to 2.4.4. The same two errors happen on all versions.

Markus,

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something inside Sphinx code, making it abort if it gets a "SEVERE" error.

Regards,
Mauro

#1 lore raw

Em Tue, 21 Apr 2020 10:38:14 +0200
Mauro Carvalho Chehab <[email protected]> escreveu:

Quoted 20 lines

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

Quoted 7 lines

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

More updates on it:

It seems that the main issue is that Sphinx passes Documentation/foo to docutils, when SPHINXDIRS=foo is used, and docutils prevent access anything outside Documentation/foo on its include and toctree directives.

So, even trying to use "../ABI/some_doc" would fail.

-

The other issue is that SphinxParallelError doesn't stop the build multithreading tasks. As Documentation/docutils.conf is set to abort on severe errors, this causes Sphinx to keep running forever, doing nothing.

I did already a workaround to prevent media Jenkins job to hang due to those issues. So, I'm moving on. Hopefully, someone with more experience with Sphinx, docutils and python multithreading can work on some fixes for it.

Thanks,
Mauro

#2 lore raw
In reply to Mauro Carvalho Chehab

Hi Mauro,

Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:

Quoted 36 lines

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad behavior, I tested (using the latest patch version) Sphinx from version 1.7.9 up to 2.4.4. The same two errors happen on all versions.

Markus,

Quoted 8 lines

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something inside Sphinx code, making it abort if it gets a "SEVERE" error.

Regards,
Mauro

I fixed it with the patch shown below. Some words about: The use of the :doc: rule should be replaced by a :ref:. The path names of include directives should always relative to the .rst file.

To go further ... with patch below:

    $ make SPHINXDIRS="driver-api" htmldocs

shows some " WARNING: undefined label: ..." messages. To close such links (addressing objects outside of the SPHINXDIR) we need to activate intersphinx [1].

If we activate intersphinx we have to wait one release cycle to get a objects.inv file at:

https://www.kernel.org/doc/html/latest/

When the objects.inv file is available, the warnings are disappear.

If the process of intersphinx is unclear, please ask.

-- Markus --

[1] https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

----

Documentation/driver-api/firewire.rst +3 -3
index 94a2d7f01d99..67741aeae5e8 100644
@@ -19,16 +19,16 @@ of kernel interfaces is available via exported symbols in `firewire-core` module
 Firewire char device data structures
 ====================================
 
-.. include:: /ABI/stable/firewire-cdev
+.. include:: ../ABI/stable/firewire-cdev
     :literal:
 
-.. kernel-doc:: include/uapi/linux/firewire-cdev.h
+.. kernel-doc:: /include/uapi/linux/firewire-cdev.h
     :internal:
 
 Firewire device probing and sysfs interfaces
 ============================================
 
-.. include:: /ABI/stable/sysfs-bus-firewire
+.. include:: ../ABI/stable/sysfs-bus-firewire
     :literal:
 
 .. kernel-doc:: drivers/firewire/core-device.c
Documentation/driver-api/gpio/using-gpio.rst +1 -1
index dda069444032..f2bd1e72af4e 100644
@@ -15,7 +15,7 @@ For any kind of mass produced system you want to support, such as servers,
 laptops, phones, tablets, routers, and any consumer or office or business goods
 using appropriate kernel drivers is paramount. Submit your code for inclusion
 in the upstream Linux kernel when you feel it is mature enough and you will get
-help to refine it, see :doc:`../../process/submitting-patches`.
+help to refine it, see :ref:`submittingpatches`.
 
 In Linux GPIO lines also have a userspace ABI.
 
Documentation/driver-api/ioctl.rst +1 -1
index c455db0e1627..ad91db68ba89 100644
@@ -34,7 +34,7 @@ _IO/_IOR/_IOW/_IOWR
 
 type
    An 8-bit number, often a character literal, specific to a subsystem
-   or driver, and listed in :doc:`../userspace-api/ioctl/ioctl-number`
+   or driver, and listed in :ref:`ioctl numbers`
 
 nr
   An 8-bit number identifying the specific command, unique for a give
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
index 52bf58417653..09647516ca52 100644
@@ -1,3 +1,5 @@
+.. _ioctl numbers:
+
 =============
 Ioctl Numbers
 =============
#3 lore raw

Hi Mauro,

one more addition: in patch

9fc3a18a94 docs: remove extra conf.py files

the conf.py from SPHINXDIRS has been removed since some latex stuff was no longer needed. IMO this is not a really good idea, this patch also remove the individual project titles like:

project = "Core-API Documentation"

and it also removes the:

tags.add("subproject")

which is widly used, e.g.:

.. only:: subproject and html

Indices
=======

* :ref:genindex

Anyway: if we do so we can also remove such directives and we can also remove the 'loadConfig' which was added in

606b9ac81a doc-rst: generic way to build only sphinx sub-folders

Sorry, that I do not have time to prepare a well formed patch. But I maintain my opinion that we'd better not do that.

-- Markus --

Am 21.04.20 um 12:52 schrieb Markus Heiser:

Quoted 138 lines

Hi Mauro,

Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:

Quoted 36 lines

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad behavior, I tested (using the latest patch version) Sphinx from version 1.7.9 up to 2.4.4. The same two errors happen on all versions.

Markus,

Quoted 8 lines

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something inside Sphinx code, making it abort if it gets a "SEVERE" error.

Regards,
Mauro

I fixed it with the patch shown below. Some words about: The use of the :doc: rule should be replaced by a :ref:. The path names of include directives should always relative to the .rst file.

To go further ... with patch below:

     $ make SPHINXDIRS="driver-api" htmldocs

shows some " WARNING: undefined label: ..." messages. To close such links (addressing objects outside of the SPHINXDIR) we need to activate intersphinx [1].

If we activate intersphinx we have to wait one release cycle to get a objects.inv file at:

https://www.kernel.org/doc/html/latest/

When the objects.inv file is available, the warnings are disappear.

If the process of intersphinx is unclear, please ask.

-- Markus --

[1] https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

----

Documentation/driver-api/firewire.rst +3 -3
index 94a2d7f01d99..67741aeae5e8 100644
@@ -19,16 +19,16 @@ of kernel interfaces is available via exported symbols in `firewire-core` module
  Firewire char device data structures
  ====================================
 
-.. include:: /ABI/stable/firewire-cdev
+.. include:: ../ABI/stable/firewire-cdev
      :literal:
 
-.. kernel-doc:: include/uapi/linux/firewire-cdev.h
+.. kernel-doc:: /include/uapi/linux/firewire-cdev.h
      :internal:
 
  Firewire device probing and sysfs interfaces
  ============================================
 
-.. include:: /ABI/stable/sysfs-bus-firewire
+.. include:: ../ABI/stable/sysfs-bus-firewire
      :literal:
 
  .. kernel-doc:: drivers/firewire/core-device.c
Documentation/driver-api/gpio/using-gpio.rst +1 -1
index dda069444032..f2bd1e72af4e 100644
@@ -15,7 +15,7 @@ For any kind of mass produced system you want to support, such as servers,
  laptops, phones, tablets, routers, and any consumer or office or business goods
  using appropriate kernel drivers is paramount. Submit your code for inclusion
  in the upstream Linux kernel when you feel it is mature enough and you will get
-help to refine it, see :doc:`../../process/submitting-patches`.
+help to refine it, see :ref:`submittingpatches`.
 
  In Linux GPIO lines also have a userspace ABI.
 
Documentation/driver-api/ioctl.rst +1 -1
index c455db0e1627..ad91db68ba89 100644
@@ -34,7 +34,7 @@ _IO/_IOR/_IOW/_IOWR
 
  type
     An 8-bit number, often a character literal, specific to a subsystem
-   or driver, and listed in :doc:`../userspace-api/ioctl/ioctl-number`
+   or driver, and listed in :ref:`ioctl numbers`
 
  nr
    An 8-bit number identifying the specific command, unique for a give
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
index 52bf58417653..09647516ca52 100644
@@ -1,3 +1,5 @@
+.. _ioctl numbers:
+
  =============
  Ioctl Numbers
  =============
#4 lore raw
In reply to Markus Heiser

Em Tue, 21 Apr 2020 12:52:42 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 44 lines

Hi Mauro,

Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:

Quoted 41 lines

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad behavior, I tested (using the latest patch version) Sphinx from version 1.7.9 up to 2.4.4. The same two errors happen on all versions.

Markus,

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something inside Sphinx code, making it abort if it gets a "SEVERE" error.

Quoted 7 lines
Quoted 3 lines

Regards,
Mauro

I fixed it with the patch shown below. Some words about: The use of the :doc: rule should be replaced by a :ref:. The path names of include directives should always relative to the .rst file.

There are lots of place already using :doc: directive, on both relative and absolute ways:

	$ git grep ':doc:'|wc -l
	382

Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits better on most needs, as it makes easier for people working with just plain texts to find to what file a ref points.

Btw, not sure why, but, when doing some conversions I got broken references when using :ref:. So, I ended using :doc: instead.

Quoted 9 lines

To go further ... with patch below:

    $ make SPHINXDIRS="driver-api" htmldocs

shows some " WARNING: undefined label: ..." messages. To close such links (addressing objects outside of the SPHINXDIR) we need to activate intersphinx [1].

Quoted 7 lines

If we activate intersphinx we have to wait one release cycle to get a objects.inv file at:

https://www.kernel.org/doc/html/latest/

When the objects.inv file is available, the warnings are disappear.

Yeah, interphinx would help a lot to solve broken references.

Not sure, however, if it is worth using it (at least for html).

I mean, IMHO, the big reason why someone would use SPHINXDIRS is to speedup the process of testing if a documentation patch did the right thing.

Anyway, it is up to Jon to decide about enabling intersphinx or not.

Quoted 15 lines

If the process of intersphinx is unclear, please ask.

-- Markus --

[1] https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

----

Documentation/driver-api/firewire.rst +0 -0
index 94a2d7f01d99..67741aeae5e8 100644

Documentation/driver-api/firewire.rst of kernel interfaces is available via exported symbols in `firewire-core` module · line 23

Quoted 6 lines
Documentation/driver-api/firewire.rst +1 -1
@@ -19,16 +19,16 @@ of kernel interfaces is available via exported symbols in `firewire-core` module
 Firewire char device data structures
 ====================================
 
-.. include:: /ABI/stable/firewire-cdev
+.. include:: ../ABI/stable/firewire-cdev
     :literal:

Hmm... I remember I tried that. It worked now on my local machine.

Yet, the big issue here is that we can't ensure that developers will always use relative paths. We need somehow convince Sphinx to always use a fixed base path (/Documentation), even when building part of a big document.

Documentation/driver-api/firewire.rst of kernel interfaces is available via exported symbols in `firewire-core` module · line 26

Quoted 3 lines
Documentation/driver-api/firewire.rst +1 -0
 -.. kernel-doc:: include/uapi/linux/firewire-cdev.h
+.. kernel-doc:: /include/uapi/linux/firewire-cdev.h
     :internal:

This is not needed. Probably because both conf.py and Makefile passes a kerneldoc source directory:

	$ grep kerneldoc_ Documentation/conf.py
	kerneldoc_bin = '../scripts/kernel-doc'
	kerneldoc_srctree = '..'
	$ grep kerneldoc_ Documentation/Makefile 
	KERNELDOC_CONF  = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
Quoted 41 lines
Documentation/driver-api/firewire.rst +1 -1
 Firewire device probing and sysfs interfaces
 ============================================
 
-.. include:: /ABI/stable/sysfs-bus-firewire
+.. include:: ../ABI/stable/sysfs-bus-firewire
     :literal:
 
 .. kernel-doc:: drivers/firewire/core-device.c
Documentation/driver-api/gpio/using-gpio.rst +1 -1
index dda069444032..f2bd1e72af4e 100644
@@ -15,7 +15,7 @@ For any kind of mass produced system you want to support, such as servers,
 laptops, phones, tablets, routers, and any consumer or office or business goods
 using appropriate kernel drivers is paramount. Submit your code for inclusion
 in the upstream Linux kernel when you feel it is mature enough and you will get
-help to refine it, see :doc:`../../process/submitting-patches`.
+help to refine it, see :ref:`submittingpatches`.
 
 In Linux GPIO lines also have a userspace ABI.
 
Documentation/driver-api/ioctl.rst +1 -1
index c455db0e1627..ad91db68ba89 100644
@@ -34,7 +34,7 @@ _IO/_IOR/_IOW/_IOWR
 
 type
    An 8-bit number, often a character literal, specific to a subsystem
-   or driver, and listed in :doc:`../userspace-api/ioctl/ioctl-number`
+   or driver, and listed in :ref:`ioctl numbers`
 
 nr
   An 8-bit number identifying the specific command, unique for a give
Documentation/userspace-api/ioctl/ioctl-number.rst +0 -0
index 52bf58417653..09647516ca52 100644

Documentation/userspace-api/ioctl/ioctl-number.rst line 5

Quoted 5 lines
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
@@ -1,3 +1,5 @@
+.. _ioctl numbers:
+
 =============
 Ioctl Numbers
 =============

Thanks,
Mauro

#5 lore raw

Hi Mauro,

about wording: I will use the term "book" for a SPHINXDIR and "bookshelf" for the whole kernel documentation ..

Am 22.04.20 um 10:57 schrieb Mauro Carvalho Chehab:

Quoted 55 lines

Em Tue, 21 Apr 2020 12:52:42 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 44 lines

Hi Mauro,

Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:

Quoted 41 lines

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad behavior, I tested (using the latest patch version) Sphinx from version 1.7.9 up to 2.4.4. The same two errors happen on all versions.

Markus,

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something inside Sphinx code, making it abort if it gets a "SEVERE" error.

Quoted 7 lines
Quoted 3 lines

Regards,
Mauro

I fixed it with the patch shown below. Some words about: The use of the :doc: rule should be replaced by a :ref:. The path names of include directives should always relative to the .rst file.

Quoted 5 lines

There are lots of place already using :doc: directive, on both relative and absolute ways:

	$ git grep ':doc:'|wc -l
	382

Your grep is wrong (it also matches the :doc: from kernel-include), use:

    $ git grep ':doc:`'|wc -l
    83
Quoted 4 lines

Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits better on most needs, as it makes easier for people working with just plain texts to find to what file a ref points.

The opposite of pointing to a file is, that we can only point the file not a content in the file and we lost the flexibility of dynamical referencing by target's name: If the target is moved, the refs has to be changed also. Not used right now, but we also lost the ability to link from one "book" to another "book" in the "bookshelf" or even into other books available in the WEB (this is what intersphinx can do).

Its a decission what counts more.

Quoted 2 lines

Btw, not sure why, but, when doing some conversions I got broken references when using :ref:. So, I ended using :doc: instead.

Can't believe, there must have been other reasons.

Quoted 12 lines
Quoted 12 lines

To go further ... with patch below:

     $ make SPHINXDIRS="driver-api" htmldocs

shows some " WARNING: undefined label: ..." messages. To close such links (addressing objects outside of the SPHINXDIR) we need to activate intersphinx [1].

If we activate intersphinx we have to wait one release cycle to get a objects.inv file at:

Quoted 8 lines
Quoted 4 lines

https://www.kernel.org/doc/html/latest/

When the objects.inv file is available, the warnings are disappear.

Yeah, interphinx would help a lot to solve broken references.

Not sure, however, if it is worth using it (at least for html).

Why should a http link does not work in a PDF file? .. I mean, its just inserting HTTP links.

Quoted 3 lines

I mean, IMHO, the big reason why someone would use SPHINXDIRS is to speedup the process of testing if a documentation patch did the right thing.

I can't speak for other, but I often prefer a book with links in over a complete bookshelf holding in my hands ;)

Quoted 1 line

Anyway, it is up to Jon to decide about enabling intersphinx or not.

-- Markus --

#7 lore raw
In reply to Mauro Carvalho Chehab

Am 22.04.20 um 12:05 schrieb Mauro Carvalho Chehab:
[...]

Quoted 7 lines
Quoted 4 lines

The opposite of pointing to a file is, that we can only point the file not a content in the file and we lost the flexibility of dynamical referencing by target's name: If the target is moved, the refs has to be changed also.

True, but we have a script that tracks those problems and even auto-correct most of such broken references.

[...]

Quoted 3 lines

An extension like Documentation/sphinx/automarkup.py could easily replace them by :doc:Documentation/foo, creating links on all those places at no cost.

[...]

Quoted 2 lines

Adding explicit :ref: would require manual work, and I bet some maintainers will reject adding extra markups on some files.

[...]

Quoted 9 lines

In any case, maybe one day intersphinx will have support for :doc:
types added there as well.

Without looking into the dirty details, it doesn't sound too hard to add support for it: intersphinx could simply create a :ref: type of reference for all files (like, __doc_driver_api__media__drivers__tuners). Then, any place with a :doc:/driver-api/media/drivers/tuner would be dynamically replaced by a :ref:__doc_driver_api__media__drivers__tuners at runtime.

Yes, hack over hack .. sometimes I feel back in the "DocBook century".

We now have a very confusing build process and I have to admit that I am also to blame.

I also believe that everyone has good reasons. For one solution or for the other solution. _Hack_ or _standard_, it is always a compromise that the Kernel community has to find.

I cannot rate that.

-- Markus --

#6 lore raw
In reply to Markus Heiser

Em Wed, 22 Apr 2020 11:27:50 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 64 lines

Hi Mauro,

about wording: I will use the term "book" for a SPHINXDIR and "bookshelf" for the whole kernel documentation ..

Am 22.04.20 um 10:57 schrieb Mauro Carvalho Chehab:

Quoted 58 lines

Em Tue, 21 Apr 2020 12:52:42 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 44 lines

Hi Mauro,

Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:

Quoted 41 lines

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that the "absolute" path is "Documentation/". So, include options like this:

.. include:: /ABI/stable/firewire-cdev
:literal:

would be evaluated as Documentation/ABI/stable/firewire-cdev if built without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then the "absolute" patch would be "Documentation/driver-api", causing this error:

Sphinx parallel build error:
docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that is meant to test media patches), as, due to some Sphinx bug, prevents sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad behavior, I tested (using the latest patch version) Sphinx from version 1.7.9 up to 2.4.4. The same two errors happen on all versions.

Markus,

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something inside Sphinx code, making it abort if it gets a "SEVERE" error.

Quoted 7 lines
Quoted 3 lines

Regards,
Mauro

I fixed it with the patch shown below. Some words about: The use of the :doc: rule should be replaced by a :ref:. The path names of include directives should always relative to the .rst file.

There are lots of place already using :doc: directive, on both relative and absolute ways:

Quoted 7 lines
Quoted 2 lines
	$ git grep ':doc:'|wc -l
	382  

Your grep is wrong (it also matches the :doc: from kernel-include), use:

    $ git grep ':doc:`'|wc -l
    83

Ah, true.

Quoted 8 lines
Quoted 3 lines

Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits better on most needs, as it makes easier for people working with just plain texts to find to what file a ref points.

The opposite of pointing to a file is, that we can only point the file not a content in the file and we lost the flexibility of dynamical referencing by target's name: If the target is moved, the refs has to be changed also.

True, but we have a script that tracks those problems and even auto-correct most of such broken references.

Right now, there are *lots* of references to a /Documentation.* file, without any sphinx tags:

	$ git grep Documentation/ Documentation/|grep -v ':doc:`'|grep -v ABI|grep -v binding|wc -l
	1515

An extension like Documentation/sphinx/automarkup.py could easily replace them by :doc:Documentation/foo, creating links on all those places at no cost.

Adding explicit :ref: would require manual work, and I bet some maintainers will reject adding extra markups on some files.

Quoted 4 lines

Not used right now, but we also
lost the ability to link from one "book" to another "book" in the "bookshelf" or even into other books available in the WEB (this is what intersphinx can do).

As currently we don't use intersphinx, I'm not too much concerned about what it could provide in the future.

In any case, maybe one day intersphinx will have support for :doc:
types added there as well.

Without looking into the dirty details, it doesn't sound too hard to add support for it: intersphinx could simply create a :ref: type of reference for all files (like, __doc_driver_api__media__drivers__tuners). Then, any place with a :doc:/driver-api/media/drivers/tuner would be dynamically replaced by a :ref:__doc_driver_api__media__drivers__tuners at runtime.

Quoted 6 lines

Its a decission what counts more.

Quoted 2 lines

Btw, not sure why, but, when doing some conversions I got broken references when using :ref:. So, I ended using :doc: instead.

Can't believe, there must have been other reasons.

Maybe. I tried several times on some documents, but it caused sphinx warnings about unknown cross references. Not sure why, and didn't investigated it further.

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

To go further ... with patch below:

     $ make SPHINXDIRS="driver-api" htmldocs

shows some " WARNING: undefined label: ..." messages. To close such links (addressing objects outside of the SPHINXDIR) we need to activate intersphinx [1].

If we activate intersphinx we have to wait one release cycle to get a objects.inv file at:

https://www.kernel.org/doc/html/latest/

When the objects.inv file is available, the warnings are disappear.

Quoted 6 lines
Quoted 3 lines

Yeah, interphinx would help a lot to solve broken references.

Not sure, however, if it is worth using it (at least for html).

Why should a http link does not work in a PDF file? .. I mean, its just inserting HTTP links.

The thing with PDF is that not many developers (I know) are using it.

Maybe now that we have a separate directory with just the PDF books, without the other temporary files people may start using it more.

Quoted 6 lines
Quoted 3 lines

I mean, IMHO, the big reason why someone would use SPHINXDIRS is to speedup the process of testing if a documentation patch did the right thing.

I can't speak for other, but I often prefer a book with links in over a complete bookshelf holding in my hands ;)

Well, I also often prefer reading a PDF book, when it provides a good comprehensive documentation. On most places, we're not there yet: there are too many gaps inside the documentation for it to be used as a real book. I hope someday this will change.

Thanks,
Mauro