Skip to contentSkip to conversation

Publish docs as GitLab Pages

PATCH In review Daniel P. Berrangé 4 participants 16 messages Nov 2, 2020 – Nov 9, 2020 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover Publish docs as GitLab Pages

This publishes the various docs as GitLab Pages, allowing users to browse the docs generated in CI before sending patch series to the mailing list. It also introduces a sidebar link to the original page source to show readers what file to modify if they want to contribute to the docs.

Daniel P. Berrangé (4):
gitlab: publish the docs built during CI
gitlab: force enable docs build in Fedora, Ubuntu, Debian
docs: show stdout/stderr when meson fails build test
docs: add "page source" link to sphinx documentation

 .gitlab-ci.yml                        | 18 +++++++++++++++++-
 docs/_templates/editpage.html         |  5 +++++
 docs/conf.py                          |  1 +
 docs/devel/_templates/editpage.html   |  5 +++++
 docs/interop/_templates/editpage.html |  5 +++++
 docs/meson.build                      |  7 ++++---
 docs/specs/_templates/editpage.html   |  5 +++++
 docs/system/_templates/editpage.html  |  5 +++++
 docs/tools/_templates/editpage.html   |  5 +++++
 docs/user/_templates/editpage.html    |  5 +++++
 10 files changed, 57 insertions(+), 4 deletions(-)
 create mode 100644 docs/_templates/editpage.html
 create mode 100644 docs/devel/_templates/editpage.html
 create mode 100644 docs/interop/_templates/editpage.html
 create mode 100644 docs/specs/_templates/editpage.html
 create mode 100644 docs/system/_templates/editpage.html
 create mode 100644 docs/tools/_templates/editpage.html
 create mode 100644 docs/user/_templates/editpage.html
Signature · 1 line
2.28.0
#1 lore raw

PATCH 1/4 gitlab: publish the docs built during CI

Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at

https://yourusername.gitlab.io/qemu/

---
 .gitlab-ci.yml | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
.gitlab-ci.yml +14 -0
index 3b15ae5c30..a41725d6f7 100644
@@ -423,3 +423,17 @@ check-dco:
       - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
   variables:
     GIT_DEPTH: 1000
+
+pages:
+  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
+  stage: test
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  script:
+    - mkdir public
+    - mv build/docs/index.html public/
+    - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
+  artifacts:
+    paths:
+      - public
Signature · 1 line
2.28.0
#2 lore raw

Re: [PATCH 1/4] gitlab: publish the docs built during CI

Quoted 6 lines · Daniel P. Berrangé

Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at

https://yourusername.gitlab.io/qemu/

Do we want that only for forks or also for the main repo?

For the main repo we need to keep older releases archived but accessible.

Quoted 13 lines
---
 .gitlab-ci.yml | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
.gitlab-ci.yml +0 -0
index 3b15ae5c30..a41725d6f7 100644
@@ -423,3 +423,17 @@ check-dco:
       - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
   variables:

.gitlab-ci.yml check-dco: · line 434

Quoted 10 lines
.gitlab-ci.yml +9 -0
     GIT_DEPTH: 1000
+
+pages:
+  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
+  stage: test
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  script:
+    - mkdir public

I'd rather use public/$CI_COMMIT_BRANCH, that way we can diff between 2 versions.

What about adding a manual job to clean public/$CI_COMMIT_BRANCH? (depending on push-pages success).

Quoted 6 lines
.gitlab-ci.yml +5 -0
+    - mv build/docs/index.html public/
+    - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
+  artifacts:
+    paths:
+      - public
#3 lore raw

Re: [PATCH 1/4] gitlab: publish the docs built during CI

Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Daniel P. Berrangé

Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at

Quoted 6 lines
Quoted 1 line

https://yourusername.gitlab.io/qemu/

Do we want that only for forks or also for the main repo?

For the main repo we need to keep older releases archived but accessible.

I'm not suggesting we use this as the actual hosting we direct end users to.

Instead, whatever cronjob on www.qemu.org currently builds

https://www.qemu.org/docs/master/

can simply download the published artifact from CI for the master branch, and any release tags, and unpack them on www.qemu.org host.

IOW, we avoid need to run builds on www.qemu.org server.

Quoted 28 lines
Quoted 14 lines
---
 .gitlab-ci.yml | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
.gitlab-ci.yml +0 -0
index 3b15ae5c30..a41725d6f7 100644
@@ -423,3 +423,17 @@ check-dco:
       - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
   variables:

.gitlab-ci.yml check-dco: · line 434

Quoted 10 lines
.gitlab-ci.yml +9 -0
     GIT_DEPTH: 1000
+
+pages:
+  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
+  stage: test
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  script:
+    - mkdir public

I'd rather use public/$CI_COMMIT_BRANCH, that way we can diff between 2 versions.

Quoted 2 lines

What about adding a manual job to clean public/$CI_COMMIT_BRANCH? (depending on push-pages success).

The publishing isn't additive - whatever is in the "public" dir is exactly what is visible on gitlab pages. You can't publish multiple different branches.

Essentially this "pages" job published an artifact containing the entire content, and this is unpacked on the gitlab.io service.

This is why I suggest www.qemu.org cronjob pull down the artifact for "master", and also pull down the artifacts for any release git tags, and unpack them in suitable subdirs.

Quoted 6 lines

.gitlab-ci.yml check-dco: · line 439

Quoted 6 lines
.gitlab-ci.yml +5 -0
+    - mv build/docs/index.html public/
+    - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
+  artifacts:
+    paths:
+      - public

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by Tested-by
#4 lore raw

Re: [PATCH 1/4] gitlab: publish the docs built during CI

Quoted 54 lines · Daniel P. Berrangé
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Daniel P. Berrangé

Most of the build jobs will create the sphinx documentation. If we expose this as an artifact of a "pages" job in a "public" directory, it will get published using GitLab Pages. This means a user can push a branch with docs changes to GitLab and view the results at

Quoted 6 lines
Quoted 1 line

https://yourusername.gitlab.io/qemu/

Do we want that only for forks or also for the main repo?

For the main repo we need to keep older releases archived but accessible.

I'm not suggesting we use this as the actual hosting we direct end users to.

Instead, whatever cronjob on www.qemu.org currently builds

https://www.qemu.org/docs/master/

can simply download the published artifact from CI for the master branch, and any release tags, and unpack them on www.qemu.org host.

IOW, we avoid need to run builds on www.qemu.org server.

Quoted 28 lines
Quoted 14 lines
---
 .gitlab-ci.yml | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
.gitlab-ci.yml +0 -0
index 3b15ae5c30..a41725d6f7 100644
@@ -423,3 +423,17 @@ check-dco:
       - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
   variables:

.gitlab-ci.yml check-dco: · line 434

Quoted 10 lines
.gitlab-ci.yml +9 -0
     GIT_DEPTH: 1000
+
+pages:
+  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
+  stage: test
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  script:
+    - mkdir public

I'd rather use public/$CI_COMMIT_BRANCH, that way we can diff between 2 versions.

Quoted 6 lines
Quoted 2 lines

What about adding a manual job to clean public/$CI_COMMIT_BRANCH? (depending on push-pages success).

The publishing isn't additive - whatever is in the "public" dir is exactly what is visible on gitlab pages. You can't publish multiple different branches.

Oh OK.

Then:

Quoted 19 lines

Essentially this "pages" job published an artifact containing the entire content, and this is unpacked on the gitlab.io service.

This is why I suggest www.qemu.org cronjob pull down the artifact for "master", and also pull down the artifacts for any release git tags, and unpack them in suitable subdirs.

Quoted 6 lines

.gitlab-ci.yml check-dco: · line 439

Quoted 6 lines
.gitlab-ci.yml +5 -0
+    - mv build/docs/index.html public/
+    - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
+  artifacts:
+    paths:
+      - public

Regards,
Daniel

#5 lore raw

PATCH 2/4 gitlab: force enable docs build in Fedora, Ubuntu, Debian

In reply to Daniel P. Berrangé

Meson runs a test to see if Sphinx works, and automatically disables it on error. This can lead to the CI jobs skipping docs build without maintainers noticing the problem. Use --enable-docs to force a fatal error if Sphinx doesn't work on the jobs where we expect it to be OK.

---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
.gitlab-ci.yml +3 -1
index a41725d6f7..f17bff652d 100644
@@ -80,6 +80,7 @@ build-system-ubuntu:
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --enable-docs
   artifacts:
     expire_in: 2 days
     paths:
@@ -111,6 +112,7 @@ build-system-debian:
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
       riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
     MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --enable-docs
   artifacts:
     expire_in: 2 days
     paths:
@@ -139,7 +141,7 @@ build-system-fedora:
   <<: *native_build_job_definition
   variables:
     IMAGE: fedora
-    CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
+    CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
     TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu
       xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
     MAKE_CHECK_ARGS: check-build
Signature · 1 line
2.28.0
Reviewed-by Tested-by on v1 2/4
#6 lore raw

Re: [PATCH 2/4] gitlab: force enable docs build in Fedora, Ubuntu, Debian

Quoted 9 lines · Daniel P. Berrangé

Meson runs a test to see if Sphinx works, and automatically disables it on error. This can lead to the CI jobs skipping docs build without maintainers noticing the problem. Use --enable-docs to force a fatal error if Sphinx doesn't work on the jobs where we expect it to be OK.

---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Shouldn't this patch go first in your series? Inverting 1<->2.

#7 lore raw

Re: [PATCH 2/4] gitlab: force enable docs build in Fedora, Ubuntu, Debian

Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Daniel P. Berrangé

Meson runs a test to see if Sphinx works, and automatically disables it on error. This can lead to the CI jobs skipping docs build without maintainers noticing the problem. Use --enable-docs to force a fatal error if Sphinx doesn't work on the jobs where we expect it to be OK.

Quoted 6 lines
Quoted 4 lines
---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Shouldn't this patch go first in your series? Inverting 1<->2.

Doesn't really matter. The jobs already build docs. It was simply that in creating this series I inserted a bug that broke the detection of sphinx by configure and I didn't notice and wondered why docs were not being created. So I just added this as a safety net to make it easier to detect future messups.

Quoted 4 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw

PATCH 3/4 docs: show stdout/stderr when meson fails build test

In reply to Daniel P. Berrangé

It is hard to diagnose why Sphinx fails in a CI environment, as we discard the stdout/err and just print a generic error message.

---
 docs/meson.build | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
docs/meson.build +4 -3
index 8c222f96bb..278098dd4f 100644
@@ -20,14 +20,15 @@ if sphinx_build.found()
   # version requirement). This will fail if sphinx-build is too old.
   run_command('mkdir', ['-p', tmpdir / 'sphinx'])
   run_command('touch', [tmpdir / 'sphinx/index.rst'])
-  sphinx_build_test_out = run_command(SPHINX_ARGS + [
+  sphinx_cmd = run_command(SPHINX_ARGS + [
     '-c', meson.current_source_dir(),
     '-b', 'html', tmpdir / 'sphinx',
     tmpdir / 'sphinx/out'])
-  build_docs = (sphinx_build_test_out.returncode() == 0)
+  build_docs = (sphinx_cmd.returncode() == 0)
 
   if not build_docs
-    warning('@0@ exists but it is either too old or uses too old a Python version'.format(get_option('sphinx_build')))
+    warning('@0@ exists but it is either too old or uses too old a Python version\nstdout:@1@\nstderr:@2@'.format(
+      get_option('sphinx_build'), sphinx_cmd.stdout().strip(), sphinx_cmd.stderr().strip()))
     if get_option('docs').enabled()
       error('Install a Python 3 version of python-sphinx')
     endif
Signature · 1 line
2.28.0
#9 lore raw

Re: [PATCH 3/4] docs: show stdout/stderr when meson fails build test

Quoted 5 lines · Daniel P. Berrangé

It is hard to diagnose why Sphinx fails in a CI environment, as we discard the stdout/err and just print a generic error message.

This kind of detailed-diagnostics should go into whatever meson's equivalent of config.log is. There's an argument for printing it if the user asked for --enable-docs specifically, but if we're doing the usual "see if it works, use it if it does" logic then we shouldn't be printing diagnostic logging from Sphinx (which can be pretty ugly and longwinded) in the middle of the meson output, IMHO.

thanks
-- PMM

#10 lore raw

Re: [PATCH 3/4] docs: show stdout/stderr when meson fails build test

Quoted 7 lines · Peter Maydell
Quoted 5 lines · Daniel P. Berrangé

It is hard to diagnose why Sphinx fails in a CI environment, as we discard the stdout/err and just print a generic error message.

Quoted 7 lines

This kind of detailed-diagnostics should go into whatever meson's equivalent of config.log is. There's an argument for printing it if the user asked for --enable-docs specifically, but if we're doing the usual "see if it works, use it if it does" logic then we shouldn't be printing diagnostic logging from Sphinx (which can be pretty ugly and longwinded) in the middle of the meson output, IMHO.

Turns out this is already recorded in build/meson-logs/meson-log.txt by default. We can probably publish the file as an artifact in gitlab CI to aid debugging.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw

PATCH 4/4 docs: add "page source" link to sphinx documentation

In reply to Daniel P. Berrangé

Add a link to the top of the sidebar in every docs page that takes the user back to the source code in gitlab.

---
 docs/_templates/editpage.html         | 5 +++++
 docs/conf.py                          | 1 +
 docs/devel/_templates/editpage.html   | 5 +++++
 docs/interop/_templates/editpage.html | 5 +++++
 docs/specs/_templates/editpage.html   | 5 +++++
 docs/system/_templates/editpage.html  | 5 +++++
 docs/tools/_templates/editpage.html   | 5 +++++
 docs/user/_templates/editpage.html    | 5 +++++
 8 files changed, 36 insertions(+)
 create mode 100644 docs/_templates/editpage.html
 create mode 100644 docs/devel/_templates/editpage.html
 create mode 100644 docs/interop/_templates/editpage.html
 create mode 100644 docs/specs/_templates/editpage.html
 create mode 100644 docs/system/_templates/editpage.html
 create mode 100644 docs/tools/_templates/editpage.html
 create mode 100644 docs/user/_templates/editpage.html
docs/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..4319b0f5ac
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
docs/conf.py +1 -0
index e584f68393..d40d8ff37b 100644
@@ -177,6 +177,7 @@ html_theme_options = {
 html_sidebars = {
     '**': [
         'about.html',
+        'editpage.html',
         'navigation.html',
         'searchbox.html',
     ]
docs/devel/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..a86d22bca8
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
docs/interop/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..215e562681
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
docs/specs/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..aaa468aa98
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/specs/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
docs/system/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..6586b2e257
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
docs/tools/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..2a9c8fc92b
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/tools/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
docs/user/_templates/editpage.html new file +5 -0
new file mode 100644
index 0000000000..1f5ee01e60
@@ -0,0 +1,5 @@
+<div id="editpage">
+  <ul>
+    <li><a href="https://gitlab.com/qemu-project/qemu/-/blob/master/docs/user/{{pagename}}.rst">Page source</a></li>
+  </ul>
+</div>
Signature · 1 line
2.28.0
#12 lore raw

Re: [PATCH 4/4] docs: add "page source" link to sphinx documentation

Quoted 3 lines · Daniel P. Berrangé

Add a link to the top of the sidebar in every docs page that takes the user back to the source code in gitlab.

This doesn't exactly link back to the source code, which is a bit confusing I think. Eg the 'page source' link in
https://berrange.gitlab.io/qemu/system/quickstart.html
goes to
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/quickstart.rst

which is a gitlab rendered version of the rST, not a view of the actual sources. Is it possible to link directly to an "actually the source code" view?

thanks
-- PMM

#13 lore raw

Re: [PATCH 4/4] docs: add "page source" link to sphinx documentation

Quoted 11 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrangé

Add a link to the top of the sidebar in every docs page that takes the user back to the source code in gitlab.

This doesn't exactly link back to the source code, which is a bit confusing I think. Eg the 'page source' link in
https://berrange.gitlab.io/qemu/system/quickstart.html
goes to
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/quickstart.rst

Quoted 3 lines

which is a gitlab rendered version of the rST, not a view of the actual sources. Is it possible to link directly to an "actually the source code" view?

At the top of that page there are two buttons for switching between the rendered view and source view. I don't think they're directly linkable as it is switched with javascript.

At least the user can see exactly what source filename is involved though, which was the key goal I had.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw
In reply to Daniel P. Berrangé
Quoted 11 lines · Daniel P. Berrangé

This publishes the various docs as GitLab Pages, allowing users to browse the docs generated in CI before sending patch series to the mailing list. It also introduces a sidebar link to the original page source to show readers what file to modify if they want to contribute to the docs.

Daniel P. Berrangé (4):
gitlab: publish the docs built during CI
gitlab: force enable docs build in Fedora, Ubuntu, Debian
docs: show stdout/stderr when meson fails build test
docs: add "page source" link to sphinx documentation

FYI (as commented in the first patch) this series result can be looked in Daniel's page:
https://berrange.gitlab.io/qemu/

#15 lore raw
In reply to Daniel P. Berrangé
Quoted 11 lines · Daniel P. Berrangé

This publishes the various docs as GitLab Pages, allowing users to browse the docs generated in CI before sending patch series to the mailing list. It also introduces a sidebar link to the original page source to show readers what file to modify if they want to contribute to the docs.

Daniel P. Berrangé (4):
gitlab: publish the docs built during CI
gitlab: force enable docs build in Fedora, Ubuntu, Debian
docs: show stdout/stderr when meson fails build test
docs: add "page source" link to sphinx documentation

Thanks, I'm going to queue patches 1, 2 and 4 for my next pull request.

Thomas