Publish docs as GitLab Pages
cover Publish docs as GitLab Pages
This publishes the various docs as GitLab Pages, allowing users to browse the docs generated in CI before sending patch series to the mailing list. It also introduces a sidebar link to the original page source to show readers what file to modify if they want to contribute to the docs.
Daniel P. Berrangé (4):
gitlab: publish the docs built during CI
gitlab: force enable docs build in Fedora, Ubuntu, Debian
docs: show stdout/stderr when meson fails build test
docs: add "page source" link to sphinx documentation
.gitlab-ci.yml | 18 +++++++++++++++++- docs/_templates/editpage.html | 5 +++++ docs/conf.py | 1 + docs/devel/_templates/editpage.html | 5 +++++ docs/interop/_templates/editpage.html | 5 +++++ docs/meson.build | 7 ++++--- docs/specs/_templates/editpage.html | 5 +++++ docs/system/_templates/editpage.html | 5 +++++ docs/tools/_templates/editpage.html | 5 +++++ docs/user/_templates/editpage.html | 5 +++++ 10 files changed, 57 insertions(+), 4 deletions(-) create mode 100644 docs/_templates/editpage.html create mode 100644 docs/devel/_templates/editpage.html create mode 100644 docs/interop/_templates/editpage.html create mode 100644 docs/specs/_templates/editpage.html create mode 100644 docs/system/_templates/editpage.html create mode 100644 docs/tools/_templates/editpage.html create mode 100644 docs/user/_templates/editpage.html
Signature · 1 line
2.28.0
PATCH 1/4 gitlab: publish the docs built during CI
Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at
https://yourusername.gitlab.io/qemu/
--- .gitlab-ci.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
.gitlab-ci.yml +14 -0
@@ -423,3 +423,17 @@ check-dco: - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master' variables: GIT_DEPTH: 1000 + +pages: + image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest + stage: test + needs: + - job: build-system-ubuntu + artifacts: true + script: + - mkdir public + - mv build/docs/index.html public/ + - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done + artifacts: + paths: + - public
Signature · 1 line
2.28.0
Re: [PATCH 1/4] gitlab: publish the docs built during CI
Quoted 6 lines · Daniel P. Berrangé
Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at
Do we want that only for forks or also for the main repo?
For the main repo we need to keep older releases archived but accessible.
Quoted 13 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+).gitlab-ci.yml +0
-0@@ -423,3 +423,17 @@ check-dco: - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master' variables:
.gitlab-ci.yml check-dco: · line 434
Quoted 10 lines
.gitlab-ci.yml +9
-0GIT_DEPTH: 1000 + +pages: + image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest + stage: test + needs: + - job: build-system-ubuntu + artifacts: true + script: + - mkdir public
I'd rather use public/$CI_COMMIT_BRANCH, that way we can diff between 2 versions.
What about adding a manual job to clean public/$CI_COMMIT_BRANCH? (depending on push-pages success).
Quoted 6 lines
.gitlab-ci.yml +5
-0+ - mv build/docs/index.html public/ + - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done + artifacts: + paths: + - public
Re: [PATCH 1/4] gitlab: publish the docs built during CI
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Daniel P. Berrangé
Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at
Quoted 6 lines
Quoted 1 line
Do we want that only for forks or also for the main repo?
For the main repo we need to keep older releases archived but accessible.
I'm not suggesting we use this as the actual hosting we direct end users to.
Instead, whatever cronjob on www.qemu.org currently builds
https://www.qemu.org/docs/master/
can simply download the published artifact from CI for the master branch, and any release tags, and unpack them on www.qemu.org host.
IOW, we avoid need to run builds on www.qemu.org server.
Quoted 28 lines
Quoted 14 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+).gitlab-ci.yml +0
-0@@ -423,3 +423,17 @@ check-dco: - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master' variables:.gitlab-ci.yml check-dco: · line 434
Quoted 10 lines
.gitlab-ci.yml +9
-0GIT_DEPTH: 1000 + +pages: + image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest + stage: test + needs: + - job: build-system-ubuntu + artifacts: true + script: + - mkdir publicI'd rather use public/$CI_COMMIT_BRANCH, that way we can diff between 2 versions.
Quoted 2 lines
What about adding a manual job to clean public/$CI_COMMIT_BRANCH? (depending on push-pages success).
The publishing isn't additive - whatever is in the "public" dir is exactly what is visible on gitlab pages. You can't publish multiple different branches.
Essentially this "pages" job published an artifact containing the entire content, and this is unpacked on the gitlab.io service.
This is why I suggest www.qemu.org cronjob pull down the artifact for "master", and also pull down the artifacts for any release git tags, and unpack them in suitable subdirs.
Quoted 6 lines
.gitlab-ci.yml check-dco: · line 439
Quoted 6 lines
.gitlab-ci.yml +5
-0+ - mv build/docs/index.html public/ + - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done + artifacts: + paths: + - public
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 1/4] gitlab: publish the docs built during CI
Quoted 54 lines · Daniel P. Berrangé
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Daniel P. Berrangé
Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at
Quoted 6 lines
Quoted 1 line
Do we want that only for forks or also for the main repo?
For the main repo we need to keep older releases archived but accessible.
I'm not suggesting we use this as the actual hosting we direct end users to.
Instead, whatever cronjob on www.qemu.org currently builds
https://www.qemu.org/docs/master/
can simply download the published artifact from CI for the master branch, and any release tags, and unpack them on www.qemu.org host.
IOW, we avoid need to run builds on www.qemu.org server.
Quoted 28 lines
Quoted 14 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+).gitlab-ci.yml +0
-0@@ -423,3 +423,17 @@ check-dco: - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master' variables:.gitlab-ci.yml check-dco: · line 434
Quoted 10 lines
.gitlab-ci.yml +9
-0GIT_DEPTH: 1000 + +pages: + image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest + stage: test + needs: + - job: build-system-ubuntu + artifacts: true + script: + - mkdir publicI'd rather use public/$CI_COMMIT_BRANCH, that way we can diff between 2 versions.
Quoted 6 lines
Quoted 2 lines
What about adding a manual job to clean public/$CI_COMMIT_BRANCH? (depending on push-pages success).
The publishing isn't additive - whatever is in the "public" dir is exactly what is visible on gitlab pages. You can't publish multiple different branches.
Oh OK.
Then:
Quoted 19 lines
Essentially this "pages" job published an artifact containing the entire content, and this is unpacked on the gitlab.io service.
This is why I suggest www.qemu.org cronjob pull down the artifact for "master", and also pull down the artifacts for any release git tags, and unpack them in suitable subdirs.
Quoted 6 lines
.gitlab-ci.yml check-dco: · line 439
Quoted 6 lines
.gitlab-ci.yml +5
-0+ - mv build/docs/index.html public/ + - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done + artifacts: + paths: + - publicRegards,
Daniel
PATCH 2/4 gitlab: force enable docs build in Fedora, Ubuntu, Debian
In reply to Daniel P. BerrangéMeson runs a test to see if Sphinx works, and automatically disables it on error. This can lead to the CI jobs skipping docs build without maintainers noticing the problem. Use --enable-docs to force a fatal error if Sphinx doesn't work on the jobs where we expect it to be OK.
--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
.gitlab-ci.yml +3 -1
@@ -80,6 +80,7 @@ build-system-ubuntu: TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu moxie-softmmu microblazeel-softmmu mips64el-softmmu MAKE_CHECK_ARGS: check-build + CONFIGURE_ARGS: --enable-docs artifacts: expire_in: 2 days paths:
@@ -111,6 +112,7 @@ build-system-debian: TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu MAKE_CHECK_ARGS: check-build + CONFIGURE_ARGS: --enable-docs artifacts: expire_in: 2 days paths:
@@ -139,7 +141,7 @@ build-system-fedora: <<: *native_build_job_definition variables: IMAGE: fedora - CONFIGURE_ARGS: --disable-gcrypt --enable-nettle + CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu MAKE_CHECK_ARGS: check-build
Signature · 1 line
2.28.0
Re: [PATCH 2/4] gitlab: force enable docs build in Fedora, Ubuntu, Debian
Quoted 9 lines · Daniel P. Berrangé
Meson runs a test to see if Sphinx works, and automatically disables it on error. This can lead to the CI jobs skipping docs build without maintainers noticing the problem. Use --enable-docs to force a fatal error if Sphinx doesn't work on the jobs where we expect it to be OK.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Shouldn't this patch go first in your series? Inverting 1<->2.
Re: [PATCH 2/4] gitlab: force enable docs build in Fedora, Ubuntu, Debian
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Daniel P. Berrangé
Meson runs a test to see if Sphinx works, and automatically disables it on error. This can lead to the CI jobs skipping docs build without maintainers noticing the problem. Use --enable-docs to force a fatal error if Sphinx doesn't work on the jobs where we expect it to be OK.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Shouldn't this patch go first in your series? Inverting 1<->2.
Doesn't really matter. The jobs already build docs. It was simply that in creating this series I inserted a bug that broke the detection of sphinx by configure and I didn't notice and wondered why docs were not being created. So I just added this as a safety net to make it easier to detect future messups.
Quoted 4 lines
Reviewed-by Philippe Mathieu-Daudé <[email protected]>
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 3/4 docs: show stdout/stderr when meson fails build test
In reply to Daniel P. BerrangéIt is hard to diagnose why Sphinx fails in a CI environment, as we discard the stdout/err and just print a generic error message.
--- docs/meson.build | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
docs/meson.build +4 -3
@@ -20,14 +20,15 @@ if sphinx_build.found() # version requirement). This will fail if sphinx-build is too old. run_command('mkdir', ['-p', tmpdir / 'sphinx']) run_command('touch', [tmpdir / 'sphinx/index.rst']) - sphinx_build_test_out = run_command(SPHINX_ARGS + [ + sphinx_cmd = run_command(SPHINX_ARGS + [ '-c', meson.current_source_dir(), '-b', 'html', tmpdir / 'sphinx', tmpdir / 'sphinx/out']) - build_docs = (sphinx_build_test_out.returncode() == 0) + build_docs = (sphinx_cmd.returncode() == 0) if not build_docs - warning('@0@ exists but it is either too old or uses too old a Python version'.format(get_option('sphinx_build'))) + warning('@0@ exists but it is either too old or uses too old a Python version\nstdout:@1@\nstderr:@2@'.format( + get_option('sphinx_build'), sphinx_cmd.stdout().strip(), sphinx_cmd.stderr().strip())) if get_option('docs').enabled() error('Install a Python 3 version of python-sphinx') endif
Signature · 1 line
2.28.0
Re: [PATCH 3/4] docs: show stdout/stderr when meson fails build test
Quoted 5 lines · Daniel P. Berrangé
It is hard to diagnose why Sphinx fails in a CI environment, as we discard the stdout/err and just print a generic error message.
Signed-off-by Daniel P. Berrangé <[email protected]>
This kind of detailed-diagnostics should go into whatever meson's equivalent of config.log is. There's an argument for printing it if the user asked for --enable-docs specifically, but if we're doing the usual "see if it works, use it if it does" logic then we shouldn't be printing diagnostic logging from Sphinx (which can be pretty ugly and longwinded) in the middle of the meson output, IMHO.
thanks
-- PMM
Re: [PATCH 3/4] docs: show stdout/stderr when meson fails build test
Quoted 7 lines · Peter Maydell
Quoted 5 lines · Daniel P. Berrangé
It is hard to diagnose why Sphinx fails in a CI environment, as we discard the stdout/err and just print a generic error message.
Signed-off-by Daniel P. Berrangé <[email protected]>
Quoted 7 lines
This kind of detailed-diagnostics should go into whatever meson's equivalent of config.log is. There's an argument for printing it if the user asked for --enable-docs specifically, but if we're doing the usual "see if it works, use it if it does" logic then we shouldn't be printing diagnostic logging from Sphinx (which can be pretty ugly and longwinded) in the middle of the meson output, IMHO.
Turns out this is already recorded in build/meson-logs/meson-log.txt by default. We can probably publish the file as an artifact in gitlab CI to aid debugging.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 4/4 docs: add "page source" link to sphinx documentation
In reply to Daniel P. BerrangéAdd a link to the top of the sidebar in every docs page that takes the user back to the source code in gitlab.
--- docs/_templates/editpage.html | 5 +++++ docs/conf.py | 1 + docs/devel/_templates/editpage.html | 5 +++++ docs/interop/_templates/editpage.html | 5 +++++ docs/specs/_templates/editpage.html | 5 +++++ docs/system/_templates/editpage.html | 5 +++++ docs/tools/_templates/editpage.html | 5 +++++ docs/user/_templates/editpage.html | 5 +++++ 8 files changed, 36 insertions(+) create mode 100644 docs/_templates/editpage.html create mode 100644 docs/devel/_templates/editpage.html create mode 100644 docs/interop/_templates/editpage.html create mode 100644 docs/specs/_templates/editpage.html create mode 100644 docs/system/_templates/editpage.html create mode 100644 docs/tools/_templates/editpage.html create mode 100644 docs/user/_templates/editpage.html
docs/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
docs/conf.py +1 -0
@@ -177,6 +177,7 @@ html_theme_options = { html_sidebars = { '**': [ 'about.html', + 'editpage.html', 'navigation.html', 'searchbox.html', ]
docs/devel/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
docs/interop/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
docs/specs/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/specs/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
docs/system/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
docs/tools/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/tools/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
docs/user/_templates/editpage.html new file +5 -0
@@ -0,0 +1,5 @@ +<div id="editpage"> + <ul> + <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/user/{{pagename}}.rst">Page source</a></li> + </ul> +</div>
Signature · 1 line
2.28.0
Re: [PATCH 4/4] docs: add "page source" link to sphinx documentation
Quoted 3 lines · Daniel P. Berrangé
Add a link to the top of the sidebar in every docs page that takes the user back to the source code in gitlab.
This doesn't exactly link back to the source code, which is a bit confusing I think. Eg the 'page source' link in
https://berrange.gitlab.io/qemu/system/quickstart.html
goes to
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/quickstart.rst
which is a gitlab rendered version of the rST, not a view of the actual sources. Is it possible to link directly to an "actually the source code" view?
thanks
-- PMM
Re: [PATCH 4/4] docs: add "page source" link to sphinx documentation
Quoted 11 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrangé
Add a link to the top of the sidebar in every docs page that takes the user back to the source code in gitlab.
This doesn't exactly link back to the source code, which is a bit confusing I think. Eg the 'page source' link in
https://berrange.gitlab.io/qemu/system/quickstart.html
goes to
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/quickstart.rst
Quoted 3 lines
which is a gitlab rendered version of the rST, not a view of the actual sources. Is it possible to link directly to an "actually the source code" view?
At the top of that page there are two buttons for switching between the rendered view and source view. I don't think they're directly linkable as it is switched with javascript.
At least the user can see exactly what source filename is involved though, which was the key goal I had.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 11 lines · Daniel P. Berrangé
This publishes the various docs as GitLab Pages, allowing users to browse the docs generated in CI before sending patch series to the mailing list. It also introduces a sidebar link to the original page source to show readers what file to modify if they want to contribute to the docs.
Daniel P. Berrangé (4):
gitlab: publish the docs built during CI
gitlab: force enable docs build in Fedora, Ubuntu, Debian
docs: show stdout/stderr when meson fails build test
docs: add "page source" link to sphinx documentation
FYI (as commented in the first patch) this series result can be looked in Daniel's page:
https://berrange.gitlab.io/qemu/
Quoted 11 lines · Daniel P. Berrangé
This publishes the various docs as GitLab Pages, allowing users to browse the docs generated in CI before sending patch series to the mailing list. It also introduces a sidebar link to the original page source to show readers what file to modify if they want to contribute to the docs.
Daniel P. Berrangé (4):
gitlab: publish the docs built during CI
gitlab: force enable docs build in Fedora, Ubuntu, Debian
docs: show stdout/stderr when meson fails build test
docs: add "page source" link to sphinx documentation
Thanks, I'm going to queue patches 1, 2 and 4 for my next pull request.
Thomas