Functions and data structure cross references with Sphinx
Subsystems (2)
There's one remaining major issue I noticed after the conversion of the media books to Sphinx:
While sphinx complains if a cross-reference (using :ref:) points to an undefined reference, the same doesn't happen if the reference uses :c:func: and :c:type:.
In practice, it means that, if we do some typo there, or if we forget to add the function/struct prototype (or use the wrong domain, like :cpp:), Sphinx won't generate the proper cross-reference, nor warning the user.
That's specially bad for media, as, while we're using the c domain for the kAPI and driver-specific books, we need to use the cpp domain on the uAPI book - as the c domain doesn't allow multiple declarations for syscalls, and we have multiple pages for read, write, open, close, poll and ioctl.
It would be good to have a way to run Sphinx on some "pedantic" mode or have something similar to xmlint that would be complaining about invalid c/cpp domain references.
Thanks,
Mauro
Am 01.08.2016 um 13:25 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 17 lines
There's one remaining major issue I noticed after the conversion of the media books to Sphinx:
While sphinx complains if a cross-reference (using :ref:) points to an undefined reference, the same doesn't happen if the reference uses :c:func: and :c:type:.
In practice, it means that, if we do some typo there, or if we forget to add the function/struct prototype (or use the wrong domain, like :cpp:), Sphinx won't generate the proper cross-reference, nor warning the user.
That's specially bad for media, as, while we're using the c domain for the kAPI and driver-specific books, we need to use the cpp domain on the uAPI book - as the c domain doesn't allow multiple declarations for syscalls, and we have multiple pages for read, write, open, close, poll and ioctl.
Quoted 6 lines
It would be good to have a way to run Sphinx on some "pedantic" mode or have something similar to xmlint that would be complaining about invalid c/cpp domain references.
Thanks,
Mauro
Hi Mauro,
there is a nit-picky mode [1], which could be activated by setting "nitpicky=True" in the conf.py or alternative, set "-n" to the SPHINXOPTS:
make SPHINXOPTS=-n htmldocs
Within nit-picky mode, Sphinx will warn about **all** references. This might be more then you want. For this, in the conf.py you could assemble a "nitpick_ignore" list [2]. But I think, assemble the ignore list is quite a lot of work.
[1] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpicky
[2] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpick_ignore
-- Markus
Em Fri, 5 Aug 2016 09:29:23 +0200
Markus Heiser <[email protected]> escreveu:
Quoted 34 lines
Am 01.08.2016 um 13:25 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 17 lines
There's one remaining major issue I noticed after the conversion of the media books to Sphinx:
While sphinx complains if a cross-reference (using :ref:) points to an undefined reference, the same doesn't happen if the reference uses :c:func: and :c:type:.
In practice, it means that, if we do some typo there, or if we forget to add the function/struct prototype (or use the wrong domain, like :cpp:), Sphinx won't generate the proper cross-reference, nor warning the user.
That's specially bad for media, as, while we're using the c domain for the kAPI and driver-specific books, we need to use the cpp domain on the uAPI book - as the c domain doesn't allow multiple declarations for syscalls, and we have multiple pages for read, write, open, close, poll and ioctl.
Quoted 6 lines
It would be good to have a way to run Sphinx on some "pedantic" mode or have something similar to xmlint that would be complaining about invalid c/cpp domain references.
Thanks,
MauroHi Mauro,
there is a nit-picky mode [1], which could be activated by setting "nitpicky=True" in the
conf.pyor alternative, set "-n" to the SPHINXOPTS:make SPHINXOPTS=-n htmldocs
Quoted 7 lines
Within nit-picky mode, Sphinx will warn about **all** references. This might be more then you want. For this, in the
conf.pyyou could assemble a "nitpick_ignore" list [2]. But I think, assemble the ignore list is quite a lot of work.[1] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpicky
[2] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpick_ignore
Yeah, this is what I was looking for.
We indeed want to use this option on media, but there are some things that need to be addressed. From some quick tests here, what I noticed is:
1) Sphinx will generate several references that should be safely ignored for everyone, like "enum", "u32", "int32_t", "bool", "NULL", etc;
2) the usage of cpp domain for system calls make several symbols to not match, as the cpp function prototype will generate cross references for the cpp domain, instead of using the c domain. So, we need a better way to fix it using the c domain, or convert everything to the cpp domain;
3) The references generated from the header files I'm parsing don't use the c (or cpp) domain. They're declared at the media book as a normal reference:
Documentation/media/uapi/v4l/field-order.rst:.. _v4l2-field:
and cross-referenced with:
ref:`v4l2_field <v4l2-field>`
Is there a way to change it to the c domain?
4) there are several references that, IMHO, should be nitpick-ignored only when the book is generated stand alone. For example, at the media docbooks, we have references for things like:
- pci_dev, mutex, off_t, container_of, etc - those are generic references for the symbols that every driver uses, but, as we don't have the books with those converted yet, nitpick complains. Once we have such references, they should be ignored *only* when the book is generated standalone. As those are "core" symbols, they should be already be documented, but the book was not ported from DocBook yet. Once we have everything ported to Sphinx, I would expect that they all will vanish (and, if not, IMHO, documenting them should be prioritized).
- References for subsystem-specific symbols like: spi_board_info, led_classdev_flash, i2c_adapter, etc. Those would require that the maintainers of the specific subsystems to add documentation to them, as I bet several such symbols won't be currently documented. So, even after the port, I afraid that we'll still have several such symbols missing.
To address (3), we need different sets of nitpick ignore lists.
At least in my case, I have two different procedures, depending on the time at the Kernel release cycle:
a) daily patch merge workflow
--------------------------
In any case, for (3), I don't want to see those warnings during
my daily patch handling process where I rebuild documentation for
every patch that touches a documented file. I want to see only
things like:
Documentation/media/uapi/v4l/hist-v4l2.rst:1295: WARNING: c:type reference target not found: struct v4l2_buffer
With indicates that a new patch would be introducing documentation gaps.
So, we need a way to have a per-subsystem nitpick_ignore list (or a way to pass such list via command line), for us to be able to ignore "alien" symbols that aren't part of the subsystem we're maintaining.
b) preparation for the merge window
--------------------------------
Late at the patch handling cycle, I run a very long task here that builds the media subsystem for 50+ different archs. I also check and fix smatch/sparse warnings. During such time, I would love to view the full list of missing symbols, in order to be able to handle eventual cross-subsystem wrong references (or eventually help documenting something that we use at the media subsystem).
So, I would need to use a different nitpick_ignore list.
Is there a way for us to specify the nitpick_ignore list (or a different conf.py) via command line?
Btw, I'm enclosing a patch adding several of those references that are alien to the media subsystem and currently causes nitpick complains.
doc-rst: ignore several undefined symbols in nitpick mode
using Sphinx in nitpick mode is too verbose to make it useful. So, we need to make it less verbose, in order to be able to actually use it.
Documentation/conf.py +85 -0
@@ -419,3 +419,88 @@ pdf_documents = [ # line arguments. kerneldoc_bin = '../scripts/kernel-doc' kerneldoc_srctree = '..' + +# +# It is possible to run Sphinx in nickpick mode with: +# make SPHINXOPTS=-n htmldocs +# In such case, it will complain about lots of missing references that +# 1) are just typedefs like: bool, __u32, etc; +# 2) It will complain for things like: enum, NULL; +# 3) It will complain for symbols that should be on different +# books (but currently aren't ported to ReST) +# The list below has a list of such symbols to be ignored in nitpick mode +# +nitpick_ignore = [ + ("c:func", "clock_gettime"), + ("c:func", "close"), + ("c:func", "container_of"), + ("c:func", "determine_valid_ioctls"), + ("c:func", "ERR_PTR"), + ("c:func", "ioctl"), + ("c:func", "IS_ERR"), + ("c:func", "mmap"), + ("c:func", "open"), + ("c:func", "pci_name"), + ("c:func", "poll"), + ("c:func", "PTR_ERR"), + ("c:func", "read"), + ("c:func", "release"), + ("c:func", "set"), + ("c:func", "struct fd_set"), + ("c:func", "struct pollfd"), + ("c:func", "usb_make_path"), + ("c:func", "write"), + ("c:type", "atomic_t"), + ("c:type", "bool"), + ("c:type", "buf_queue"), + ("c:type", "device"), + ("c:type", "device_driver"), + ("c:type", "device_node"), + ("c:type", "enum"), + ("c:type", "file"), + ("c:type", "i2c_adapter"), + ("c:type", "i2c_board_info"), + ("c:type", "i2c_client"), + ("c:type", "ktime_t"), + ("c:type", "led_classdev_flash"), + ("c:type", "list_head"), + ("c:type", "lock_class_key"), + ("c:type", "module"), + ("c:type", "mutex"), + ("c:type", "pci_dev"), + ("c:type", "pdvbdev"), + ("c:type", "poll_table_struct"), + ("c:type", "s32"), + ("c:type", "s64"), + ("c:type", "sd"), + ("c:type", "spi_board_info"), + ("c:type", "spi_device"), + ("c:type", "spi_master"), + ("c:type", "struct fb_fix_screeninfo"), + ("c:type", "struct pollfd"), + ("c:type", "struct timeval"), + ("c:type", "struct video_capability"), + ("c:type", "u16"), + ("c:type", "u32"), + ("c:type", "u64"), + ("c:type", "u8"), + ("c:type", "union"), + ("c:type", "usb_device"), + + ("cpp:type", "boolean"), + ("cpp:type", "fd"), + ("cpp:type", "fd_set"), + ("cpp:type", "int16_t"), + ("cpp:type", "NULL"), + ("cpp:type", "off_t"), + ("cpp:type", "pollfd"), + ("cpp:type", "size_t"), + ("cpp:type", "ssize_t"), + ("cpp:type", "timeval"), + ("cpp:type", "__u16"), + ("cpp:type", "__u32"), + ("cpp:type", "__u64"), + ("cpp:type", "uint16_t"), + ("cpp:type", "uint32_t"), + ("cpp:type", "video_system_t"), +]
Am 05.08.2016 um 12:47 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 124 lines
Em Fri, 5 Aug 2016 09:29:23 +0200
Markus Heiser <[email protected]> escreveu:Quoted 34 lines
Am 01.08.2016 um 13:25 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 17 lines
There's one remaining major issue I noticed after the conversion of the media books to Sphinx:
While sphinx complains if a cross-reference (using :ref:) points to an undefined reference, the same doesn't happen if the reference uses :c:func: and :c:type:.
In practice, it means that, if we do some typo there, or if we forget to add the function/struct prototype (or use the wrong domain, like :cpp:), Sphinx won't generate the proper cross-reference, nor warning the user.
That's specially bad for media, as, while we're using the c domain for the kAPI and driver-specific books, we need to use the cpp domain on the uAPI book - as the c domain doesn't allow multiple declarations for syscalls, and we have multiple pages for read, write, open, close, poll and ioctl.
Quoted 6 lines
It would be good to have a way to run Sphinx on some "pedantic" mode or have something similar to xmlint that would be complaining about invalid c/cpp domain references.
Thanks,
MauroHi Mauro,
there is a nit-picky mode [1], which could be activated by setting "nitpicky=True" in the
conf.pyor alternative, set "-n" to the SPHINXOPTS:make SPHINXOPTS=-n htmldocs
Quoted 7 lines
Within nit-picky mode, Sphinx will warn about **all** references. This might be more then you want. For this, in the
conf.pyyou could assemble a "nitpick_ignore" list [2]. But I think, assemble the ignore list is quite a lot of work.[1] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpicky
[2] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpick_ignoreYeah, this is what I was looking for.
We indeed want to use this option on media, but there are some things that need to be addressed. From some quick tests here, what I noticed is:
1) Sphinx will generate several references that should be safely ignored for everyone, like "enum", "u32", "
int32_t", "bool", "NULL", etc;2) the usage of cpp domain for system calls make several symbols to not match, as the cpp function prototype will generate cross references for the cpp domain, instead of using the c domain. So, we need a better way to fix it using the c domain, or convert everything to the cpp domain;
3) The references generated from the header files I'm parsing don't use the c (or cpp) domain. They're declared at the media book as a normal reference:
Documentation/media/uapi/v4l/field-order.rst:.. _v4l2-field:and cross-referenced with:
ref:`v4l2_field <v4l2-field>`Is there a way to change it to the c domain?
4) there are several references that, IMHO, should be nitpick-ignored only when the book is generated stand alone. For example, at the media docbooks, we have references for things like:
-
pci_dev, mutex,off_t,container_of, etc - those are generic references for the symbols that every driver uses, but, as we don't have the books with those converted yet, nitpick complains. Once we have such references, they should be ignored *only* when the book is generated standalone. As those are "core" symbols, they should be already be documented, but the book was not ported from DocBook yet. Once we have everything ported to Sphinx, I would expect that they all will vanish (and, if not, IMHO, documenting them should be prioritized).- References for subsystem-specific symbols like:
spi_board_info,led_classdev_flash,i2c_adapter, etc. Those would require that the maintainers of the specific subsystems to add documentation to them, as I bet several such symbols won't be currently documented. So, even after the port, I afraid that we'll still have several such symbols missing.To address (3), we need different sets of nitpick ignore lists.
At least in my case, I have two different procedures, depending on the time at the Kernel release cycle:
a) daily patch merge workflow
--------------------------In any case, for (3), I don't want to see those warnings during
my daily patch handling process where I rebuild documentation for
every patch that touches a documented file. I want to see only
things like:
Documentation/media/uapi/v4l/hist-v4l2.rst:1295: WARNING: c:type reference target not found: structv4l2_bufferWith indicates that a new patch would be introducing documentation gaps.
So, we need a way to have a per-subsystem
nitpick_ignorelist (or a way to pass such list via command line), for us to be able to ignore "alien" symbols that aren't part of the subsystem we're maintaining.b) preparation for the merge window
--------------------------------Late at the patch handling cycle, I run a very long task here that builds the media subsystem for 50+ different archs. I also check and fix smatch/sparse warnings. During such time, I would love to view the full list of missing symbols, in order to be able to handle eventual cross-subsystem wrong references (or eventually help documenting something that we use at the media subsystem).
Quoted 4 lines
So, I would need to use a different
nitpick_ignorelist.Is there a way for us to specify the
nitpick_ignorelist (or a differentconf.py) via command line?
Since conf.py is python, we could implement something, which loads a "conditional configuration", e.g. loading a config with a nitpick_ignore list in. Depending on an environment
"SPHINX_BUILD_CONF=[strong|lazy]-nit-picking.py"
we could load individual build-configs overwriting the default settings from the origin conf.py.
On which repo/branch you are working? .. I send you a RFC patch for "conditional configurations".
Back to (3) ... as far as I know, there is no way to add a *Internal Hyperlink Target* (e.g. "_v4l2-field:") to the C or CPP domain.
There is a ":any:" directive does something vice versa.
http://www.sphinx-doc.org/en/stable/markup/inline.html#role-any
But I think, this will not help referencing a type from a function prototype to a *Internal Hyperlink Target*, like the struct described under the "_v4l2-field:" target.
If ":any:" does not help, we might find a solution with an additional crossref-type or something similar:
http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_crossref_type
But this needs some more thoughts.
-- Markus --
Quoted 108 lines
Btw, I'm enclosing a patch adding several of those references that are alien to the media subsystem and currently causes nitpick complains.
doc-rst: ignore several undefined symbols in nitpick mode
using Sphinx in nitpick mode is too verbose to make it useful. So, we need to make it less verbose, in order to be able to actually use it.
Signed-off-by Mauro Carvalho Chehab <[email protected]>Documentation/conf.py +0
-0@@ -419,3 +419,88 @@ pdf_documents = [ # line arguments.
kerneldoc_bin= '../scripts/kernel-doc'kerneldoc_srctree= '..'Documentation/conf.py +85
-0+ +# +# It is possible to run Sphinx in nickpick mode with: +# make SPHINXOPTS=-n htmldocs +# In such case, it will complain about lots of missing references that +# 1) are just typedefs like: bool, __u32, etc; +# 2) It will complain for things like: enum, NULL; +# 3) It will complain for symbols that should be on different +# books (but currently aren't ported to ReST) +# The list below has a list of such symbols to be ignored in nitpick mode +# +nitpick_ignore = [ + ("c:func", "clock_gettime"), + ("c:func", "close"), + ("c:func", "container_of"), + ("c:func", "determine_valid_ioctls"), + ("c:func", "ERR_PTR"), + ("c:func", "ioctl"), + ("c:func", "IS_ERR"), + ("c:func", "mmap"), + ("c:func", "open"), + ("c:func", "pci_name"), + ("c:func", "poll"), + ("c:func", "PTR_ERR"), + ("c:func", "read"), + ("c:func", "release"), + ("c:func", "set"), + ("c:func", "struct fd_set"), + ("c:func", "struct pollfd"), + ("c:func", "usb_make_path"), + ("c:func", "write"), + ("c:type", "atomic_t"), + ("c:type", "bool"), + ("c:type", "buf_queue"), + ("c:type", "device"), + ("c:type", "device_driver"), + ("c:type", "device_node"), + ("c:type", "enum"), + ("c:type", "file"), + ("c:type", "i2c_adapter"), + ("c:type", "i2c_board_info"), + ("c:type", "i2c_client"), + ("c:type", "ktime_t"), + ("c:type", "led_classdev_flash"), + ("c:type", "list_head"), + ("c:type", "lock_class_key"), + ("c:type", "module"), + ("c:type", "mutex"), + ("c:type", "pci_dev"), + ("c:type", "pdvbdev"), + ("c:type", "poll_table_struct"), + ("c:type", "s32"), + ("c:type", "s64"), + ("c:type", "sd"), + ("c:type", "spi_board_info"), + ("c:type", "spi_device"), + ("c:type", "spi_master"), + ("c:type", "struct fb_fix_screeninfo"), + ("c:type", "struct pollfd"), + ("c:type", "struct timeval"), + ("c:type", "struct video_capability"), + ("c:type", "u16"), + ("c:type", "u32"), + ("c:type", "u64"), + ("c:type", "u8"), + ("c:type", "union"), + ("c:type", "usb_device"), + + ("cpp:type", "boolean"), + ("cpp:type", "fd"), + ("cpp:type", "fd_set"), + ("cpp:type", "int16_t"), + ("cpp:type", "NULL"), + ("cpp:type", "off_t"), + ("cpp:type", "pollfd"), + ("cpp:type", "size_t"), + ("cpp:type", "ssize_t"), + ("cpp:type", "timeval"), + ("cpp:type", "__u16"), + ("cpp:type", "__u32"), + ("cpp:type", "__u64"), + ("cpp:type", "uint16_t"), + ("cpp:type", "uint32_t"), + ("cpp:type", "video_system_t"), +]
Em Fri, 5 Aug 2016 14:22:19 +0200
Markus Heiser <[email protected]> escreveu:
Quoted 9 lines
Am 05.08.2016 um 12:47 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 6 lines
Em Fri, 5 Aug 2016 09:29:23 +0200
Markus Heiser <[email protected]> escreveu:Is there a way for us to specify the
nitpick_ignorelist (or a differentconf.py) via command line?
Quoted 8 lines
Since
conf.pyis python, we could implement something, which loads a "conditional configuration", e.g. loading a config with anitpick_ignorelist in. Depending on an environment"SPHINX_BUILD_CONF=[strong|lazy]-nit-picking.py"
we could load individual build-configs overwriting the default settings from the origin
conf.py.
That would be interesting!
Quoted 2 lines
On which repo/branch you are working? .. I send you a RFC patch for "conditional configurations".
You can send it against either the media_tree.git[1] or the upstream one[2]. All Sphinx patches I have were merged there already (except for the experimental nickpick patch I attached on the previous e-mail).
[1] https://git.linuxtv.org/media_tree.git/log/
[2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/
Quoted 9 lines
Quoted 5 lines
3) The references generated from the header files I'm parsing don't use the c (or cpp) domain. They're declared at the media book as a normal reference:
Documentation/media/uapi/v4l/field-order.rst:.. _v4l2-field:Back to (3) ... as far as I know, there is no way to add a *Internal Hyperlink Target* (e.g. "_v4l2-field:") to the C or CPP domain.
Argh!
Quoted 8 lines
There is a ":any:" directive does something vice versa.
http://www.sphinx-doc.org/en/stable/markup/inline.html#role-any
But I think, this will not help referencing a type from a function prototype to a *Internal Hyperlink Target*, like the struct described under the "_v4l2-field:" target.
Hmm... it might help, but this is Sphinx 1.3 or upper only. As we decide to set the minimal version bar to 1.2, we should avoid using it.
Quoted 7 lines
If ":any:" does not help, we might find a solution with an additional crossref-type or something similar:
http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_crossref_type
But this needs some more thoughts.
That sounds more promising, but we'll need a replacement for the :c:func: tag to use it.
Thanks,
Mauro