parts of media docs sphinx re-building every time?
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 13 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.
Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.
Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed for one feature ... not very straight forward.
If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
Quoted 2 lines
(I presume
kernel_include.py
is mostly a workaround to keep out-of-tree builds working?)
Yes, e.g. with "make O=/tmp/kernel htmldocs" the parse-headers.pl output goes to /tmp/kernel and is included by ".. kernel-include: $BUILDDIR/xxx"
-- Markus --
Quoted 8 lines
Anyway, the rebuild part is most important. This must be fixed.
BR,
Jani.Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
Hi Jani,
Em Mon, 8 Aug 2016 18:07:10 +0200
Markus Heiser <[email protected]> escreveu:
Quoted 9 lines
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 5 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
Perhaps there are some Makefile dependencies there that are not ok. I'll look in to it.
Quoted 13 lines
Quoted 8 lines
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.
Quoted 7 lines
Yes, parse-headers.pl,
kernel_include.pyand media/Makefile are needed for one feature ... not very straight forward.If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
As discussed before, we need to be able to auto-generate cross references from the headers. Unfortunately, Sphinx acts like a 5-years-old-boy by painting source files some random colors, but not doing anything useful like creating cross references with the documentation.
So, we need an extra script for the media build to convert the API headers into rst files. This work is somewhat complex, as there are symbols that we explicitly want to ignore, including ifdef symbols like:
#define _UAPI__LINUX_VIDEODEV2_H #ifdef _UAPI__LINUX_VIDEODEV2_H ... #endif
We also want to do things like:
replace symbol V4L2_TUNER_ANALOG_TV v4l2-tuner-type replace symbol V4L2_TUNER_RADIO v4l2-tuner-type replace symbol V4L2_TUNER_RF v4l2-tuner-type replace symbol V4L2_TUNER_SDR v4l2-tuner-type
in order to make all symbols to point to the same element at the rst file, that are usually inside a table.
(I actually want to change this to point to an specific row at the table, but there are almost 400 symbols to be fixed, and changing it will take some time, and will likely require manual work).
The goal of Documentation/sphinx/parse-headers.pl script is to generate such parsed headers, with the cross-references modified by an exceptions file at Documentation/media/*.h.rst.exceptions.
This returns back a feature that we used to have with DocBook.
The Documentation/media/Makefile rules what should be converted, and what exception file will be used to generate the rst file:
$(BUILDDIR)/audio.h.rst: ${UAPI}/dvb/audio.h ${PARSER} $(SRC_DIR)/audio.h.rst.exceptions
@$($(quiet)gen_rst)We might move that to Documentation/Makefile.sphinx, if you don't like having another makefile, but IMHO, this will be messy and will cause conflicts during the merge window.
With regards to use python, well... I don't program on python, nor I'm interested on doing it ATM... I actually wrote one python script a long time ago - that I had to fix to work on a newer python 2.x version, as the unicode API was changed - and very likely it won't work on python 3 anymore, as lots of API got changed.
The thing is: perl is reliable enough for not needing to rewrite the script every time someone comes with some crazy idea that would break the language API and force changes at the scripts. So, I prefer to keep that script in a language that doesn't bite me on upgrades. As a plus, it doesn't forces me to adopt random alien code style of 4 space indentations, and not use tabs. But that's me.
So, I'm not against porting it. Yet, what would be the advantage of porting it to Python? If there's no clear advantage, let's keep it in perl, as it is easier to maintain.
Thanks,
Mauro
Quoted 7 lines · Jani Nikula
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.
Quoted 5 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly. (I presumekernel_include.pyis mostly a workaround to keep out-of-tree builds working?)
Additionally, 'make pdfdocs' fails with e.g.
/path/to/linux/Documentation/media/uapi/cec/cec-header.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/cec.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/audio_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/audio.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/ca_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/ca.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/dmx_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/dmx.h.rst'.
because the makefile hack is only done on htmldocs target.
BR,
Jani.
Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
Quoted 3 lines · Mauro Carvalho Chehab
The goal of
Documentation/sphinx/parse-headers.plscript is to generate such parsed headers, with the cross-references modified by an exceptions file atDocumentation/media/*.h.rst.exceptions.
Would you be so kind as to state in a few lines what you want to achieve? I can guess based on the current solution, but I'd like to hear it from you. Please leave out rants about tools and languages etc. so we can focus on the problem statement, and try to figure out the best overall solution.
Thanks,
Jani.
Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
Quoted 22 lines · Markus Heiser
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 13 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.
Quoted 7 lines
Yes, parse-headers.pl,
kernel_include.pyand media/Makefile are needed for one feature ... not very straight forward.If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
If I understand the need of all of this right, I think the cleanest and fastest short term measure would be to make the kernel-include directive extension do the same thing as the kernel-doc directive does: call the perl script from the directive.
This lets you get rid of Documentation/media/Makefile and you don't have to copy-paste all of Include.run method into kernel_include.py. You can also get rid of specifying environment variables in rst files and parsing them in the extension. We can get rid of the problematic intermediate rst files. This design has been proven with the kernel-doc extension and script already. It's much simpler.
BR,
Jani.
Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
Quoted 10 lines · On Wed, 10 Aug 2016 12:23:16 +0300 Jani Nikula <[email protected]> wrote:
Quoted 5 lines
Quoted 2 lines
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I was unable to reproduce it here. Are you passing any special options to the building system?
Hmh, I can't reproduce this now either. I was able to hit this on another machine consistently, even with 'make cleandocs' in between. I'll check the environment on the other machine when I get my hands on it.
Just FWIW, I've been trying to find a moment to come back to this because I couldn't reproduce it either...
jon
Quoted 10 lines · Mauro Carvalho Chehab
Em Mon, 08 Aug 2016 18:37:38 +0300
Jani Nikula <[email protected]> escreveu:Quoted 4 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I was unable to reproduce it here. Are you passing any special options to the building system?
Hmh, I can't reproduce this now either. I was able to hit this on another machine consistently, even with 'make cleandocs' in between. I'll check the environment on the other machine when I get my hands on it.
BR,
Jani.
Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
Em Wed, 10 Aug 2016 10:42:27 +0300
Jani Nikula <[email protected]> escreveu:
Quoted 25 lines · Jani Nikula
Quoted 7 lines
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 5 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly. (I presumekernel_include.pyis mostly a workaround to keep out-of-tree builds working?)Additionally, 'make pdfdocs' fails with e.g.
/path/to/linux/Documentation/media/uapi/cec/cec-header.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/cec.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/audio_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/audio.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/ca_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/ca.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/dmx_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/dmx.h.rst'.
Quoted 1 line
because the makefile hack is only done on htmldocs target.
Hit send too early... this is what happens here if I run make with the documents already built:
$ make htmldocs make BUILDDIR=Documentation/output -f ./Documentation/media/Makefile htmldocs make[2]: Nothing to be done for 'htmldocs'. SPHINX htmldocs Running Sphinx v1.4.5 loading pickled environment... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 0 source files that are out of date updating environment: 0 added, 0 changed, 0 removed looking for now-outdated files... none found no targets are out of date. build succeeded.
(docbook build messages suppressed
Cheers,
Mauro
Em Mon, 08 Aug 2016 18:37:38 +0300
Jani Nikula <[email protected]> escreveu:
Quoted 4 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I was unable to reproduce it here. Are you passing any special options to the building system?
According to media Makefile, it should run the script only on four conditions:
- if the dynamically-generated rst file is not found (e. g. after make cleandocs);
- if the header file is changed;
- if the exceptions file changes;
- if the perl parser is changed.
All rules are like:
$(BUILDDIR)/audio.h.rst: ${UAPI}/dvb/audio.h ${PARSER} $(SRC_DIR)/audio.h.rst.exceptions
Regards,
Mauro
Cheers,
Mauro
Em Wed, 10 Aug 2016 11:58:48 +0200
Markus Heiser <[email protected]> escreveu:
Quoted 50 lines
Am 10.08.2016 um 11:22 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 48 lines
Em Wed, 10 Aug 2016 12:15:34 +0300
Jani Nikula <[email protected]> escreveu:Quoted 36 lines · Markus Heiser
Quoted 22 lines
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 13 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.Quoted 7 lines
Yes, parse-headers.pl,
kernel_include.pyand media/Makefile are needed for one feature ... not very straight forward.If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
If I understand the need of all of this right, I think the cleanest and fastest short term measure would be to make the kernel-include directive extension do the same thing as the kernel-doc directive does: call the perl script from the directive.
Quoted 6 lines
This lets you get rid of
Documentation/media/Makefileand you don't have to copy-paste all of Include.run method intokernel_include.py. You can also get rid of specifying environment variables in rst files and parsing them in the extension. We can get rid of the problematic intermediate rst files. This design has been proven with the kernel-doc extension and script already. It's much simpler.Works for me. If someone comes with such patch, I'll happily ack it.
Quoted 7 lines
Quoted 2 lines
Cheers,
MauroHi Jani & Mauro,
I will give it a try ... but currently I'am working in some other tasks. I think next week I will find some time to implement.
Good enough to me, Thanks for looking into that!
Cheers,
Mauro
Am 10.08.2016 um 11:22 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 46 lines
Em Wed, 10 Aug 2016 12:15:34 +0300
Jani Nikula <[email protected]> escreveu:Quoted 36 lines · Markus Heiser
Quoted 22 lines
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 13 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.Quoted 7 lines
Yes, parse-headers.pl,
kernel_include.pyand media/Makefile are needed for one feature ... not very straight forward.If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
If I understand the need of all of this right, I think the cleanest and fastest short term measure would be to make the kernel-include directive extension do the same thing as the kernel-doc directive does: call the perl script from the directive.
Quoted 6 lines
This lets you get rid of
Documentation/media/Makefileand you don't have to copy-paste all of Include.run method intokernel_include.py. You can also get rid of specifying environment variables in rst files and parsing them in the extension. We can get rid of the problematic intermediate rst files. This design has been proven with the kernel-doc extension and script already. It's much simpler.
Quoted 4 lines
Works for me. If someone comes with such patch, I'll happily ack it.
Cheers,
Mauro
Hi Jani & Mauro,
I will give it a try ... but currently I'am working in some other tasks. I think next week I will find some time to implement.
-- Markus --
Em Wed, 10 Aug 2016 12:15:34 +0300
Jani Nikula <[email protected]> escreveu:
Quoted 36 lines · Markus Heiser
Quoted 22 lines
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 13 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.Quoted 7 lines
Yes, parse-headers.pl,
kernel_include.pyand media/Makefile are needed for one feature ... not very straight forward.If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
If I understand the need of all of this right, I think the cleanest and fastest short term measure would be to make the kernel-include directive extension do the same thing as the kernel-doc directive does: call the perl script from the directive.
Quoted 6 lines
This lets you get rid of
Documentation/media/Makefileand you don't have to copy-paste all of Include.run method intokernel_include.py. You can also get rid of specifying environment variables in rst files and parsing them in the extension. We can get rid of the problematic intermediate rst files. This design has been proven with the kernel-doc extension and script already. It's much simpler.
Works for me. If someone comes with such patch, I'll happily ack it.
Cheers,
Mauro
Em Wed, 10 Aug 2016 10:46:44 +0300
Jani Nikula <[email protected]> escreveu:
Quoted 10 lines · Mauro Carvalho Chehab
Quoted 3 lines
The goal of
Documentation/sphinx/parse-headers.plscript is to generate such parsed headers, with the cross-references modified by an exceptions file atDocumentation/media/*.h.rst.exceptions.Would you be so kind as to state in a few lines what you want to achieve? I can guess based on the current solution, but I'd like to hear it from you. Please leave out rants about tools and languages etc. so we can focus on the problem statement, and try to figure out the best overall solution.
It is basically what's written above: to produce a cross-referenced output document from a source, were, clicking on the API symbols will navigate to the place where the symbol was documented.
It should allow to add exceptions, as we don't want to add cross-references for legacy symbols or helper macros, for example.
This is a good example on what we want:
https://linuxtv.org/downloads/v4l-dvb-apis-new/media/uapi/rc/lirc-header.html#lirc-h
The only parts of the header file that will have a different color will be the symbols defined at the API.
This way, it is easy for us to visually discover what stuff is not yet documented (like LIRC_MODE2_SPACE - that should be documented - and _LINUX_LIRC_H macro - that should not be documented).
When something is not cross-referenced there, there's a single place to look that will explain why this was not documented (at the exceptions file).
The main goal for documentation writers is to use the header
file to identify the documentation gaps. For code developers, the header file works like an index to the document, where all the API can be seen altogether, and more details can easily be obtained by clicking at the links.
Regard
Cheers,
Mauro
Am 10.08.2016 um 15:46 schrieb Jonathan Corbet <[email protected]>:
Quoted 14 lines
Quoted 10 lines · On Wed, 10 Aug 2016 12:23:16 +0300 Jani Nikula <[email protected]> wrote:
Quoted 5 lines
Quoted 2 lines
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I was unable to reproduce it here. Are you passing any special options to the building system?
Hmh, I can't reproduce this now either. I was able to hit this on another machine consistently, even with 'make cleandocs' in between. I'll check the environment on the other machine when I get my hands on it.
Quoted 4 lines
Just FWIW, I've been trying to find a moment to come back to this because I couldn't reproduce it either...
jon
Hmm, I have had problems with the relative BUILDDIR make environment, so I switched to absolute pathname .. see my "more generic way" patch:
htmldocs:
- $(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
+ $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) -f $(srctree)/Documentation/media/Makefile $@
could this the reason why you can't reproduce it?
My problem was vice versa, if I called "make O=/tmp/kernel htmldocs" after a make with normal output, the rst files has been found in Documents/output and not regenerated in /tmp/kernel/Documents/output.
And with "make O=/tmp/kernel clean", the rst files in Documents/output resists.
This was very confusing to me, so I changed it to absolute pathname.
--Markus--
Quoted 36 lines · Jani Nikula
Quoted 22 lines · Markus Heiser
Hi Jani,
Am 08.08.2016 um 17:37 schrieb Jani Nikula <[email protected]>:
Quoted 13 lines
Hi Mauro & co -
I just noticed running 'make htmldocs' rebuilds parts of media docs every time on repeated runs. This shouldn't happen. Please investigate.
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 4 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly.Quoted 7 lines
Yes, parse-headers.pl,
kernel_include.pyand media/Makefile are needed for one feature ... not very straight forward.If it makes sense to migrate the perl scripts functionality to a Sphinx extension, may I can help ... depends on what Mauro thinks.
BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
If I understand the need of all of this right, I think the cleanest and fastest short term measure would be to make the kernel-include directive extension do the same thing as the kernel-doc directive does: call the perl script from the directive.
Quoted 6 lines
This lets you get rid of
Documentation/media/Makefileand you don't have to copy-paste all of Include.run method intokernel_include.py. You can also get rid of specifying environment variables in rst files and parsing them in the extension. We can get rid of the problematic intermediate rst files. This design has been proven with the kernel-doc extension and script already. It's much simpler.
I looked a bit at this and seems interesting ... a few questions:
- Are you using this just for uapi headers or also for other bits?
- My concern with out-of-line docs is always that people forget to update them. How do you enforce that in the media subsystem?
- Atm we don't have any formal way to document drm ioctl, and this could be a possible approach. Would it be possible to share this with other subsystems, maybe extended/polished, perhaps even as the official way to document uapi headers?
Just some thoughts, orthogonal to the discussion at hand here. -Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Em Wed, 10 Aug 2016 10:42:27 +0300
Jani Nikula <[email protected]> escreveu:
Quoted 25 lines · Jani Nikula
Quoted 7 lines
I wonder if it's related to
Documentation/media/Makefile... which I have to say I am not impressed by. I was really hoping we could build all the documentation by standalone sphinx-build invocation too, relying only on theconf.pyso that e.g. Read the Docs can build the docs. Part of that motivation was to keep the build clean in makefiles, and handing the dependency tracking completely to Sphinx.Quoted 5 lines
I believe what's in
Documentation/media/Makefile,Documentation/sphinx/parse-headers.pl, andDocumentation/sphinx/kernel_include.pycould be replaced by a Sphinx extension looking at the sources directly. (I presumekernel_include.pyis mostly a workaround to keep out-of-tree builds working?)Additionally, 'make pdfdocs' fails with e.g.
/path/to/linux/Documentation/media/uapi/cec/cec-header.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/cec.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/audio_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/audio.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/ca_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/ca.h.rst'. /path/to/linux/Documentation/media/uapi/dvb/dmx_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/dmx.h.rst'.
Quoted 1 line
because the makefile hack is only done on htmldocs target.
Yeah, we need to call the media Makefile on other targets. I'll see if I can write a quick fix for that. Please notice, however, that I'm on a trip this week, with not much time available for writing patches.
Yet, you probably missed some e-mails while you were in vacations. rst2pdf is not capable of handling complex documents. It crashes with the media docs. See this patch:
https://patchwork.kernel.org/patch/9231687/
There is a thread about that too... but, after checking my backlogs, I noticed that this was not c/c to the ML, with is a shame :(
I'm enclosing the part of the thread where we've discussed about rst2pdf, that resulted on the above patch.
In summary:
- rst2pdf is completely broken with Sphinx 1.3.x;
- rst2pdf doesn't handle complex documents, even with Sphinx 1.4.5;
- rst2pdf always report a successful build, even when it fails;
- Markus recommended removing the media book from the build, as he doesn't know any fixup or replacement for the tool.
That's why I came up with that patch, and recommended to even disable pdf production at the patch comments while we don't have a sane alternative.
Regards,
Mauro
Am 13.07.2016 um 17:41 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 39 lines
Em Wed, 13 Jul 2016 17:22:05 +0200
Markus Heiser <[email protected]> escreveu:Quoted 36 lines
Am 13.07.2016 um 14:40 schrieb Mauro Carvalho Chehab <[email protected]>:
Quoted 20 lines
Em Thu, 7 Jul 2016 10:12:36 +0200
Markus Heiser <[email protected]> escreveu:With Sphinx version 1.4.4/1.4.5, no errors were produced from media DocBook, with is great.
Yet, trying to use pdfdocs produce this error:
ERROR] pdfbuilder.py:130 too many values to unpack
Traceback (most recent call last):
File "/home/mchehab/.local/lib/python2.7/site-packages/rst2pdf/pdfbuilder.py", line 122, in write
appendices=opts.get('pdf_appendices', self.config.pdf_appendices) or [])
File "/home/mchehab/.local/lib/python2.7/site-packages/rst2pdf/pdfbuilder.py", line 209, inassemble_doctree
index_nodes=genindex_nodes(genindex)
File "/home/mchehab/.local/lib/python2.7/site-packages/rst2pdf/pdfbuilder.py", line 385, ingenindex_nodes
for entryname, (links, subitems) in entries:
ValueError: too many values to unpack
FAILED
build succeeded.Quoted 2 lines
(for rst2pdf to work, I had to use pip2, as the version it downloaded doesn't run with python3)
There was a time, I thought this rst2pdf tool could be a solution which only needs some bug fixes, but in the meantime I have to say, that rst2pdf is "broken by design".
This is very sad, but after dig into rst2pdf to fix some bugs I had to realized that it has tons of access on non existing properties.
https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542
Getting a build without an exception is just a game of luck. :-(
Quoted 7 lines
Quoted 1 line
Sorry for the bad news ...
Gah, that sucks. It sucks even more that sphinx is uncapable of detecting that something got wrong, keep saying that:
"build succedded"and keeping the zero byte pdf stored there.
It is not only sphinx, it is the rst2pdf extension which does not throw a SystemMessagePropagation exception on fatal errors.
Quoted 1 line
Are there any other alternative to produce pdfs?
Yes, the sphinx-doc build-in LaTeX builder
* http://www.sphinx-doc.org/en/stable/config.html#options-for-latex-output
But it has some drawbacks, e.g. it produce LaTeX for the pdfTeX engine. LaTeX is by default ASCII and it needs some "inputenc" to supportma wider range of characters. This is not very helpful if you have a toolchain in an international community.
The alternative to LaTeX is to use the XeTeX engine, which supports UTF-8 encoded input by default and supports TrueType/OpenType fonts directly. Thats why I started to write a XeLaTeX builder ...
* https://github.com/return42/sphkerneldoc/blob/master/scripts/site-python/xelatex_ext/__init__.py#L15
... but I can't predict when this will be finished ...
However which tool is used, my experience is, that building PDF (books) with a minimum of quality is not simple. Layout width tables, split table content over pages, switch from landscape to portrait and versus, the flow of objects etc. .. all this will need some manually interventions.
Quoted 4 lines
I suspect that Jani and Jon may not be happy by the fact that the media book causes pdf generation to break. So, maybe we'll need some sort of "magic" to make sphinx to exclude the build for the media docbook, if pdf was requested.
No magic, only structure ;-)
in the conf.py you will find the settings (similar to man page settings)
pdf_documents = [
('index', u'Kernel', u'Kernel', u'J. Random Bozo'),
]
With "index", the PDF is build for the root node, means all books and text files in one big PDF ... IMHO (see above) this will never work.
Since we have this structure:
.. toctree::
:maxdepth: 2
kernel-documentation
media/media_uapi
I recommend to edit the pdf_documents value to:
pdf_documents = [
('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
]
this should prevent building PDF from media book
-- Markus --
Signature · 2 lines
Cheers, Mauro