Skip to contentSkip to conversation

sphinx: Use separate doctree directories for different builders

PATCH In review Eduardo Habkost 3 participants 3 messages Oct 14, 2019 – Oct 17, 2019 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sphinx: Use separate doctree directories for different builders

sphinx-build is buggy when multiple processes are using the same doctree directory in parallel. See the 3-year-old Sphinx bug report at: https://github.com/sphinx-doc/sphinx/issues/2946

Instead of avoiding parallel builds or adding some kind of locking, I'm using the simplest solution: just using a different doctree cache for each builder.

---
 Makefile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Makefile +4 -1
index 30f0abfb42..d20e7ffce3 100644
@@ -983,7 +983,10 @@ sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index
 
 # Canned command to build a single manual
 # Arguments: $1 = manual name, $2 = Sphinx builder ('html' or 'man')
-build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
+# Note the use of different doctree for each (manual, builder) tuple;
+# this works around Sphinx not handling parallel invocation on
+# a single doctree: https://github.com/sphinx-doc/sphinx/issues/2946
+build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1-$2 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
 # We assume all RST files in the manual's directory are used in it
 manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
 
Signature · 1 line
2.21.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 10 lines · Eduardo Habkost

sphinx-build is buggy when multiple processes are using the same doctree directory in parallel. See the 3-year-old Sphinx bug report at: https://github.com/sphinx-doc/sphinx/issues/2946

Instead of avoiding parallel builds or adding some kind of locking, I'm using the simplest solution: just using a different doctree cache for each builder.

Quoted 11 lines
---
 Makefile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Makefile +0 -0
index 30f0abfb42..d20e7ffce3 100644
@@ -983,7 +983,10 @@ sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index
 

Makefile sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index · line 992

Quoted 10 lines
Makefile +4 -1
 # Canned command to build a single manual
 # Arguments: $1 = manual name, $2 = Sphinx builder ('html' or 'man')
-build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
+# Note the use of different doctree for each (manual, builder) tuple;
+# this works around Sphinx not handling parallel invocation on
+# a single doctree: https://github.com/sphinx-doc/sphinx/issues/2946
+build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1-$2 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
 # We assume all RST files in the manual's directory are used in it
 manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
 

Thanks for this!

#2 lore raw
In reply to Eduardo Habkost
Quoted 9 lines · Eduardo Habkost

sphinx-build is buggy when multiple processes are using the same doctree directory in parallel. See the 3-year-old Sphinx bug report at: https://github.com/sphinx-doc/sphinx/issues/2946

Instead of avoiding parallel builds or adding some kind of locking, I'm using the simplest solution: just using a different doctree cache for each builder.

Quoted 5 lines
---
 Makefile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Applied to master as a buildfix, thanks.

-- PMM