improve output when building docs
cover improve output when building docs
When building the documentation, a lot of text is output, meaning that any warnings can be missed in all the text. Unfortunately, ninja merges both stderr and stdout of all tasks so one cannot just redirect stdout to a separate location as part of the build command to rectify that. Therefore, since we rarely care about the output of the doc builds, only output the stderr text and write the standard output text to a log file in the relevant build folder (i.e. build/doc/guides or build/doc/api)
Bruce Richardson (4):
doc/api: hide verbose doxygen standard output doc/api: align output folder with sphinx guides doc/api: put output log file in build directory doc/guides: suppress printing out standard output
buildtools/call-sphinx-build.py | 6 ++++-- doc/api/generate_doxygen.sh | 6 ++++-- doc/api/meson.build | 6 +++--- 3 files changed, 11 insertions(+), 7 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/4 doc/api: hide verbose doxygen standard output
The standard output of doxygen is very verbose, and since ninja mixes stdout and stderr together it makes it difficult to see any warnings from the doxygen run. Therefore, we can just log the standard output to file, and only output the stderr to make warnings clear.
--- doc/api/generate_doxygen.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
doc/api/generate_doxygen.sh +1 -1
@@ -7,7 +7,7 @@ OUTDIR=$2 SCRIPTCSS=$3 # run doxygen, capturing all the header files it processed -doxygen "${DOXYCONF}" | tee doxygen.out +doxygen "${DOXYCONF}" > doxygen.out echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' doxygen.out)" > $OUTDIR.d "${SCRIPTCSS}" "${OUTDIR}"/doxygen.css
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH 1/4] doc/api: hide verbose doxygen standard output
Quoted 4 lines
The standard output of doxygen is very verbose, and since ninja mixes stdout and stderr together it makes it difficult to see any warnings from the doxygen run. Therefore, we can just log the standard output to file, and only output the stderr to make warnings clear.
[...]
doc/api/generate_doxygen.sh OUTDIR=$2 · line 10
Quoted 2 lines
doc/api/generate_doxygen.sh +1
-1-doxygen "${DOXYCONF}" | tee doxygen.out +doxygen "${DOXYCONF}" > doxygen.out
ninja is printing extra lines:
ninja: Entering directory `build'
[0/1] Running external command doc
Building docs: Doxygen_API HTML_Guides
It makes hard to detect whether all run fine or not. Can we remove these extra lines with a ninja option? Can we return an error if there are some errors in doxygen?
Re: [dpdk-dev] [PATCH 1/4] doc/api: hide verbose doxygen standard output
Quoted 13 lines · Thomas Monjalon
Quoted 4 lines
The standard output of doxygen is very verbose, and since ninja mixes stdout and stderr together it makes it difficult to see any warnings from the doxygen run. Therefore, we can just log the standard output to file, and only output the stderr to make warnings clear.
[...]
doc/api/generate_doxygen.sh OUTDIR=$2 · line 10
Quoted 2 lines
doc/api/generate_doxygen.sh +1
-1-doxygen "${DOXYCONF}" | tee doxygen.out +doxygen "${DOXYCONF}" > doxygen.outninja is printing extra lines:
ninja: Entering directory `build'
[0/1] Running external command doc
Building docs:Doxygen_APIHTML_Guides
Quoted 4 lines
It makes hard to detect whether all run fine or not. Can we remove these extra lines with a ninja option? Can we return an error if there are some errors in doxygen?
If doxygen fails, then the whole build should fail - though to be honest that is hard to test since most invalid changes I've tried making end up just as warnings. The wrapper script for doxygen has "-e" flag so it should fail if any part of it does, i.e. doxygen, and ninja always returns error if any subtask fails.
There is also the WARN_AS_ERROR setting for doxygen which we can use if you want to be stricter [1]. Once all warnings are fixed, I'd suggest setting that based on the werror setting for the build as a whole.
[1] https://www.doxygen.nl/manual/config.html#cfg_warn_as_error
Re: [dpdk-dev] [PATCH 1/4] doc/api: hide verbose doxygen standard output
29/09/2020 18:23, Bruce Richardson:
Quoted 24 lines · Thomas Monjalon
Quoted 13 lines
Quoted 4 lines
The standard output of doxygen is very verbose, and since ninja mixes stdout and stderr together it makes it difficult to see any warnings from the doxygen run. Therefore, we can just log the standard output to file, and only output the stderr to make warnings clear.
[...]
doc/api/generate_doxygen.sh OUTDIR=$2 · line 10
Quoted 2 lines
doc/api/generate_doxygen.sh +1
-1-doxygen "${DOXYCONF}" | tee doxygen.out +doxygen "${DOXYCONF}" > doxygen.outninja is printing extra lines:
ninja: Entering directory `build'
[0/1] Running external command doc
Building docs:Doxygen_APIHTML_GuidesQuoted 4 lines
It makes hard to detect whether all run fine or not. Can we remove these extra lines with a ninja option? Can we return an error if there are some errors in doxygen?
If doxygen fails, then the whole build should fail - though to be honest that is hard to test since most invalid changes I've tried making end up just as warnings. The wrapper script for doxygen has "-e" flag so it should fail if any part of it does, i.e. doxygen, and ninja always returns error if any subtask fails.
Quoted 5 lines
There is also the
WARN_AS_ERRORsetting for doxygen which we can use if you want to be stricter [1]. Once all warnings are fixed, I'd suggest setting that based on the werror setting for the build as a whole.[1] https://www.doxygen.nl/manual/config.html#cfg_warn_as_error
Yes good idea.
Can we make WARN_AS_ERROR depending on the meson option -werror?
Re: [dpdk-dev] [PATCH 1/4] doc/api: hide verbose doxygen standard output
Quoted 25 lines · Thomas Monjalon
29/09/2020 18:23, Bruce Richardson:
Quoted 24 lines · Thomas Monjalon
Quoted 13 lines
Quoted 4 lines
The standard output of doxygen is very verbose, and since ninja mixes stdout and stderr together it makes it difficult to see any warnings from the doxygen run. Therefore, we can just log the standard output to file, and only output the stderr to make warnings clear.
[...]
doc/api/generate_doxygen.sh OUTDIR=$2 · line 10
Quoted 2 lines
doc/api/generate_doxygen.sh +1
-1-doxygen "${DOXYCONF}" | tee doxygen.out +doxygen "${DOXYCONF}" > doxygen.outninja is printing extra lines:
ninja: Entering directory `build'
[0/1] Running external command doc
Building docs:Doxygen_APIHTML_GuidesQuoted 4 lines
It makes hard to detect whether all run fine or not. Can we remove these extra lines with a ninja option? Can we return an error if there are some errors in doxygen?
If doxygen fails, then the whole build should fail - though to be honest that is hard to test since most invalid changes I've tried making end up just as warnings. The wrapper script for doxygen has "-e" flag so it should fail if any part of it does, i.e. doxygen, and ninja always returns error if any subtask fails.
Quoted 9 lines
Quoted 5 lines
There is also the
WARN_AS_ERRORsetting for doxygen which we can use if you want to be stricter [1]. Once all warnings are fixed, I'd suggest setting that based on the werror setting for the build as a whole.[1] https://www.doxygen.nl/manual/config.html#cfg_warn_as_error
Yes good idea.
Can we makeWARN_AS_ERRORdepending on the meson option -werror?
Yes, very easily, because the doxygen config file is adjustable from meson. If you want I can do a v2 with a fix for doxygen ethdev and the setting for WARN_AS_ERROR added as extra patches to this.
PATCH 2/4 doc/api: align output folder with sphinx guides
In reply to Bruce RichardsonThe API docs were output to "<build>/doc/api/api" folder, which was ugly-looking with the repeated "api", and inconsistent with the sphinx guides which were written to "<build>/doc/guides/html". Changing the doxgen output folder to "html" fixes both these issues.
--- doc/api/meson.build | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
doc/api/meson.build +3 -3
@@ -34,7 +34,7 @@ cdata = configuration_data() cdata.set('VERSION', meson.project_version()) cdata.set('API_EXAMPLES', join_paths(meson.build_root(), 'doc', 'api', 'examples.dox')) cdata.set('OUTPUT', join_paths(meson.build_root(), 'doc', 'api')) -cdata.set('HTML_OUTPUT', 'api') +cdata.set('HTML_OUTPUT', 'html') cdata.set('TOPDIR', meson.source_root()) cdata.set('STRIP_FROM_PATH', meson.source_root())
@@ -45,8 +45,8 @@ doxy_conf = configure_file(input: 'doxy-api.conf.in', doxy_build = custom_target('doxygen', depends: example, input: doxy_conf, - output: 'api', - depfile: 'api.d', + output: 'html', + depfile: 'html.d', command: [generate_doxygen, '@INPUT@', '@OUTPUT@', generate_css], install: get_option('enable_docs'), install_dir: htmldir,
Signature · 1 line
2.25.1
PATCH 3/4 doc/api: put output log file in build directory
In reply to Bruce RichardsonThe meson documentation states that projects should not rely upon the custom_target build commands are run from any given directory. Therefore, rather than writing the standout output from doxygen to the current directory - which could be anywhere in future, put it into the api directory, so that it is in a known location.
--- doc/api/generate_doxygen.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
doc/api/generate_doxygen.sh +4 -2
@@ -6,8 +6,10 @@ DOXYCONF=$1 OUTDIR=$2 SCRIPTCSS=$3 +OUT_FILE=$(dirname $OUTDIR)/doxygen.out + # run doxygen, capturing all the header files it processed -doxygen "${DOXYCONF}" > doxygen.out -echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' doxygen.out)" > $OUTDIR.d +doxygen "${DOXYCONF}" > $OUT_FILE +echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' $OUT_FILE)" > $OUTDIR.d "${SCRIPTCSS}" "${OUTDIR}"/doxygen.css
Signature · 1 line
2.25.1
PATCH 4/4 doc/guides: suppress printing out standard output
In reply to Bruce RichardsonTo see only errors and warnings from the doc builds, we can send the standard output text to a logfile and have only the stderr messages printed. This is similar to what is done for the API documentation.
--- buildtools/call-sphinx-build.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
buildtools/call-sphinx-build.py +4 -2
@@ -27,8 +27,10 @@ srcfiles.extend([join(root, f) for f in files]) # run sphinx, putting the html output in a "html" directory -process = run(sphinx_cmd + ['-b', 'html', src, join(dst, 'html')], check=True) -print(str(process.args) + ' Done OK') +with open(join(dst, 'sphinx_html.out'), 'w') as out: + process = run(sphinx_cmd + ['-b', 'html', src, join(dst, 'html')], + check=True, + stdout=out) # create a gcc format .d file giving all the dependencies of this doc build with open(join(dst, '.html.d'), 'w') as d:
Signature · 1 line
2.25.1
v2 cover improve building docs
In reply to Bruce RichardsonWhen building the documentation, a lot of text is output, meaning that any warnings can be missed in all the text. Unfortunately, ninja merges both stderr and stdout of all tasks so one cannot just redirect stdout to a separate location as part of the build command to rectify that. Therefore, since we rarely care about the output of the doc builds, only output the stderr text and write the standard output text to a log file in the relevant build folder (i.e. build/doc/guides or build/doc/api)
Related to this, we can catch documentation bugs earlier by setting WARN_AS_ERRORS for doxygen, meaning the build will fail if any API documentation warnings as encountered. Rather than setting this globally, we can link it to the global build --werror setting.
---
V2: added in patches to fix API doc issue, and enable WARN_AS_ERRORS
Bruce Richardson (6):
doc/api: hide verbose doxygen standard output doc/api: align output folder with sphinx guides doc/api: put output log file in build directory doc/guides: suppress printing out standard output
ethdev: fix mis-named parameter
doc/api: make doc warnings errors when werror option set
buildtools/call-sphinx-build.py | 6 ++++-- doc/api/doxy-api.conf.in | 1 + doc/api/generate_doxygen.sh | 6 ++++-- doc/api/meson.build | 10 +++++++--- lib/librte_ethdev/rte_ethdev.h | 2 +- 5 files changed, 17 insertions(+), 8 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/6 doc/api: hide verbose doxygen standard output
The standard output of doxygen is very verbose, and since ninja mixes stdout and stderr together it makes it difficult to see any warnings from the doxygen run. Therefore, we can just log the standard output to file, and only output the stderr to make warnings clear.
--- doc/api/generate_doxygen.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
doc/api/generate_doxygen.sh +1 -1
@@ -7,7 +7,7 @@ OUTDIR=$2 SCRIPTCSS=$3 # run doxygen, capturing all the header files it processed -doxygen "${DOXYCONF}" | tee doxygen.out +doxygen "${DOXYCONF}" > doxygen.out echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' doxygen.out)" > $OUTDIR.d "${SCRIPTCSS}" "${OUTDIR}"/doxygen.css
Signature · 1 line
2.25.1
PATCH v2 2/6 doc/api: align output folder with sphinx guides
In reply to Bruce RichardsonThe API docs were output to "<build>/doc/api/api" folder, which was ugly-looking with the repeated "api", and inconsistent with the sphinx guides which were written to "<build>/doc/guides/html". Changing the doxygen output folder to "html" fixes both these issues.
--- doc/api/meson.build | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
doc/api/meson.build +3 -3
@@ -34,7 +34,7 @@ cdata = configuration_data() cdata.set('VERSION', meson.project_version()) cdata.set('API_EXAMPLES', join_paths(meson.build_root(), 'doc', 'api', 'examples.dox')) cdata.set('OUTPUT', join_paths(meson.build_root(), 'doc', 'api')) -cdata.set('HTML_OUTPUT', 'api') +cdata.set('HTML_OUTPUT', 'html') cdata.set('TOPDIR', meson.source_root()) cdata.set('STRIP_FROM_PATH', meson.source_root())
@@ -45,8 +45,8 @@ doxy_conf = configure_file(input: 'doxy-api.conf.in', doxy_build = custom_target('doxygen', depends: example, input: doxy_conf, - output: 'api', - depfile: 'api.d', + output: 'html', + depfile: 'html.d', command: [generate_doxygen, '@INPUT@', '@OUTPUT@', generate_css], install: get_option('enable_docs'), install_dir: htmldir,
Signature · 1 line
2.25.1
PATCH v2 3/6 doc/api: put output log file in build directory
In reply to Bruce RichardsonThe meson documentation states that projects should not rely upon the custom_target build commands are run from any given directory. Therefore, rather than writing the standout output from doxygen to the current directory - which could be anywhere in future, put it into the api directory, so that it is in a known location.
--- doc/api/generate_doxygen.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
doc/api/generate_doxygen.sh +4 -2
@@ -6,8 +6,10 @@ DOXYCONF=$1 OUTDIR=$2 SCRIPTCSS=$3 +OUT_FILE=$(dirname $OUTDIR)/doxygen.out + # run doxygen, capturing all the header files it processed -doxygen "${DOXYCONF}" > doxygen.out -echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' doxygen.out)" > $OUTDIR.d +doxygen "${DOXYCONF}" > $OUT_FILE +echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' $OUT_FILE)" > $OUTDIR.d "${SCRIPTCSS}" "${OUTDIR}"/doxygen.css
Signature · 1 line
2.25.1
PATCH v2 4/6 doc/guides: suppress printing out standard output
In reply to Bruce RichardsonTo see only errors and warnings from the doc builds, we can send the standard output text to a logfile and have only the stderr messages printed. This is similar to what is done for the API documentation.
--- buildtools/call-sphinx-build.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
buildtools/call-sphinx-build.py +4 -2
@@ -27,8 +27,10 @@ srcfiles.extend([join(root, f) for f in files]) # run sphinx, putting the html output in a "html" directory -process = run(sphinx_cmd + ['-b', 'html', src, join(dst, 'html')], check=True) -print(str(process.args) + ' Done OK') +with open(join(dst, 'sphinx_html.out'), 'w') as out: + process = run(sphinx_cmd + ['-b', 'html', src, join(dst, 'html')], + check=True, + stdout=out) # create a gcc format .d file giving all the dependencies of this doc build with open(join(dst, '.html.d'), 'w') as d:
Signature · 1 line
2.25.1
PATCH v2 5/6 ethdev: fix mis-named parameter
In reply to Bruce RichardsonThe parameter to rte_eth_link_speed_to_str was called "speed_link" in the function prototype in the header file, but "link_speed" everywhere else. This showed up as warnings when building the API docs, due to missing and undocumented parameters from doxygen's viewpoint.
Rename the prototype value to the correct name to fix these issues.
--- lib/librte_ethdev/rte_ethdev.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_ethdev/rte_ethdev.h +1 -1
@@ -2452,7 +2452,7 @@ int rte_eth_link_get_nowait(uint16_t port_id, struct rte_eth_link *link);* No free is required. */ __rte_experimental -const char *rte_eth_link_speed_to_str(uint32_t speed_link); +const char *rte_eth_link_speed_to_str(uint32_t link_speed);/** * @warning
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 5/6] ethdev: fix mis-named parameter
Quoted 6 lines
The parameter to
rte_eth_link_speed_to_strwas called "speed_link" in the function prototype in the header file, but "link_speed" everywhere else. This showed up as warnings when building the API docs, due to missing and undocumented parameters from doxygen's viewpoint.Rename the prototype value to the correct name to fix these issues.
It has been fixed by David and merged already by Ferruh.
Re: [dpdk-dev] [PATCH v2 5/6] ethdev: fix mis-named parameter
Quoted 9 lines · Thomas Monjalon
Quoted 6 lines
The parameter to
rte_eth_link_speed_to_strwas called "speed_link" in the function prototype in the header file, but "link_speed" everywhere else. This showed up as warnings when building the API docs, due to missing and undocumented parameters from doxygen's viewpoint.Rename the prototype value to the correct name to fix these issues.
It has been fixed by David and merged already by Ferruh.
Yep, I guessed there was a patch outstanding for it from someone, though I think the patch wasn't merged when I did this patch, so I included it to guarantee that this patchset didn't break the build when werror was turned on.
/Bruce
Re: [dpdk-dev] [PATCH v2 5/6] ethdev: fix mis-named parameter
In reply to Bruce RichardsonQuoted 6 lines · Bruce Richardson
The parameter to
rte_eth_link_speed_to_strwas called "speed_link" in the function prototype in the header file, but "link_speed" everywhere else. This showed up as warnings when building the API docs, due to missing and undocumented parameters from doxygen's viewpoint.
Quoted 6 lines
Rename the prototype value to the correct name to fix these issues.
Fixes fbf931c9c392 ("ethdev: format link status text")Signed-off-by Bruce Richardson <[email protected]>
Already fixed with:
https://git.dpdk.org/next/dpdk-next-net/commit/?id=10c835f2f57a62e50c83c764926d2481afbb60d5
Signature · 1 line
David Marchand
PATCH v2 6/6 doc/api: make doc warnings errors when werror option set
In reply to Bruce RichardsonWhen the --werror meson build option is set, we can set the WARN_AS_ERRORS doxygen option in the doxygen config flag to get the same behaviour for API doc building as for building the rest of DPDK. This can help catch documentation errors sooner in the development process.
--- doc/api/doxy-api.conf.in | 1 + doc/api/meson.build | 4 ++++ 2 files changed, 5 insertions(+)
doc/api/doxy-api.conf.in +1 -0
@@ -89,6 +89,7 @@ HIDE_SCOPE_NAMES = YES GENERATE_DEPRECATEDLIST = YES VERBATIM_HEADERS = NO ALPHABETICAL_INDEX = NO +WARN_AS_ERROR = @WARN_AS_ERROR@ HTML_TIMESTAMP = NO HTML_DYNAMIC_SECTIONS = YES
doc/api/meson.build +4 -0
@@ -37,6 +37,10 @@ cdata.set('OUTPUT', join_paths(meson.build_root(), 'doc', 'api')) cdata.set('HTML_OUTPUT', 'html') cdata.set('TOPDIR', meson.source_root()) cdata.set('STRIP_FROM_PATH', meson.source_root()) +cdata.set('WARN_AS_ERROR', 'NO') +if get_option('werror') + cdata.set('WARN_AS_ERROR', 'YES') +endif doxy_conf = configure_file(input: 'doxy-api.conf.in', output: 'doxy-api.conf',
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 6/6] doc/api: make doc warnings errors when werror option set
doc/api/doxy-api.conf.in +1 -0
@@ -89,6 +89,7 @@ HIDE_SCOPE_NAMES = YES GENERATE_DEPRECATEDLIST = YES VERBATIM_HEADERS = NO ALPHABETICAL_INDEX = NO +WARN_AS_ERROR = @WARN_AS_ERROR@
This section is about the appearance of the output.
I think this new "warning" option would better fit at the end of the file.
Re: [dpdk-dev] [PATCH v2 0/6] improve building docs
In reply to Bruce RichardsonQuoted 8 lines
When building the documentation, a lot of text is output, meaning that any warnings can be missed in all the text. Unfortunately, ninja merges both stderr and stdout of all tasks so one cannot just redirect stdout to a separate location as part of the build command to rectify that. Therefore, since we rarely care about the output of the doc builds, only output the stderr text and write the standard output text to a log file in the relevant build folder (i.e. build/doc/guides or build/doc/api)
Quoted 4 lines
Related to this, we can catch documentation bugs earlier by setting
WARN_AS_ERRORSfor doxygen, meaning the build will fail if any API documentation warnings as encountered. Rather than setting this globally, we can link it to the global build --werror setting.
Applied with suggested minor change, and without ethdev fix. Thanks