Skip to contentSkip to conversation

docs: Use make invocation's -j argument for parallelism

PATCH Superseded Kees Cook 4 participants 7 messages Sep 19, 2019 – Sep 25, 2019 Documentation

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Kees Cook

author
#0 lore raw

PATCH v2 1/1 docs: Use make invocation's -j argument for parallelism

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available parallelism from "make"'s job server (since it is not exposed in any special variables) and use that for the "sphinx-build" run. Now things work correctly for builds where -j is specified at the top-level:

make -j16 htmldocs

If -j is not specified, continue to fallback to "-jauto" if available.

---
v2: retain "-jauto" default behavior with top-level -j is missing.

---
 Documentation/Makefile  |  3 ++-
 scripts/jobserver-count | 53 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)
 create mode 100755 scripts/jobserver-count
Documentation/Makefile +2 -1
index e145e4db508b..8bfd38a865ff 100644
@@ -33,7 +33,7 @@ ifeq ($(HAVE_SPHINX),0)
 
 else # HAVE_SPHINX
 
-export SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version 2>&1 |"; while (<IN>) { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN')
+export SPHINX_PARALLEL = $(shell perl -e 'open IN,"sphinx-build --version 2>&1 |"; while (<IN>) { if (m/([\d\.]+)/) { print "auto" if ($$1 >= "1.7") } ;} close IN')
 
 # User-friendly check for pdflatex and latexmk
 HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
@@ -68,6 +68,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
 	PYTHONDONTWRITEBYTECODE=1 \
 	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
 	$(SPHINXBUILD) \
+	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \
 	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
scripts/jobserver-count new file +53 -0

53 lines not on this page · raw message

Signature · 5 lines
2.17.1


-- 
Kees Cook
maintainer
#1 lore raw
Quoted 3 lines · On Thu, 19 Sep 2019 14:44:37 -0700 Kees Cook <[email protected]> wrote:

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available

What sort of confusion might we expect? Or, to channel akpm, "what are the user-visible effects of this bug"?

Quoted 7 lines

parallelism from "make"'s job server (since it is not exposed in any special variables) and use that for the "sphinx-build" run. Now things work correctly for builds where -j is specified at the top-level:

make -j16 htmldocs

If -j is not specified, continue to fallback to "-jauto" if available.

So this seems like a good thing to do. I do have a couple of small issues, though...

[...]

Documentation/Makefile quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) · line 71

Quoted 1 line
Documentation/Makefile +1 -0
+	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \

This (and the shebang line in the script itself) will cause the docs build to fail on systems lacking Python 3. While we have talked about requiring Python 3 for the docs build, we have not actually taken that step yet. We probably shouldn't sneak it in here. I don't see anything in the script that should require a specific Python version, so I think it should be tweaked to be version-independent and just invoke "python".

scripts/jobserver-count line 2

Quoted 11 lines
Documentation/Makefile +0 -0
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \
 	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
scripts/jobserver-count new file +2 -0
new file mode 100755
index 000000000000..ff6ebe6b0194
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later

By license-rules.rst, this should be GPL-2.0+

Quoted 19 lines
scripts/jobserver-count new file +19 -0
+#
+# This determines how many parallel tasks "make" is expecting, as it is
+# not exposed via an special variables.
+# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+import os, sys, fcntl
+
+# Default parallelism is "1" unless overridden on the command-line.
+default="1"
+if len(sys.argv) > 1:
+	default=sys.argv[1]
+
+# Set non-blocking for a given file descriptor.
+def nonblock(fd):
+	flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+	fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+	return fd
+
+# Extract and prepare jobserver file descriptors from envirnoment.
+try:

scripts/jobserver-count line 31

Quoted 10 lines
scripts/jobserver-count new file +10 -0
+	# Fetch the make environment options.
+	flags = os.environ['MAKEFLAGS']
+
+	# Look for "--jobserver=R,W"
+	opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
+	# Parse out R,W file descriptor numbers and set them nonblocking.
+	fds = opts[0].split("=", 1)[1]
+	reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
+except:

So I have come to really dislike bare "except" clauses; I've seen them hide too many bugs. In this case, perhaps it's justified, but still ... it bugs me ...

scripts/jobserver-count line 43

Quoted 12 lines
scripts/jobserver-count new file +12 -0
+	# Any failures here should result in just using the default
+	# specified parallelism.
+	print(default)
+	sys.exit(0)
+
+# Read out as many jobserver slots as possible.
+jobs = b""
+while True:
+	try:
+		slot = os.read(reader, 1)
+		jobs += slot
+	except:

This one, I think, should be explicit; anything other than EWOULDBLOCK indicates a real problem, right?

scripts/jobserver-count line 46

Quoted 3 lines
scripts/jobserver-count new file +3 -0
+		break
+# Return all the reserved slots.
+os.write(writer, jobs)

You made writer nonblocking, so it seems plausible that we could leak some slots here, no? Does writer really need to be nonblocking?

scripts/jobserver-count line 53

Quoted 6 lines
scripts/jobserver-count new file +6 -0
+# If the jobserver was (impossibly) full or communication failed, use default.
+if len(jobs) < 1:
+	print(default)
+
+# Report available slots (with a bump for our caller's reserveration).
+print(len(jobs) + 1)

The last question I have is...why is it that we have to do this complex dance rather than just passing the "-j" option through directly to sphinx? That comes down to the "confusion" mentioned at the top, I assume. It would be good to understand that?

Thanks,

jon

author
#2 lore raw
Quoted 9 lines · Jonathan Corbet
Quoted 3 lines · On Thu, 19 Sep 2019 14:44:37 -0700 Kees Cook <[email protected]> wrote:

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available

What sort of confusion might we expect? Or, to channel akpm, "what are the user-visible effects of this bug"?

When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not parallelized. When I run "make htmldocs -j8" with 1.7+ sphinx, it uses all my CPUs instead of 8. :)

Quoted 8 lines

Documentation/Makefile quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) · line 71

Quoted 1 line
Documentation/Makefile +1 -0
+	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \

This (and the shebang line in the script itself) will cause the docs build to fail on systems lacking Python 3. While we have talked about requiring Python 3 for the docs build, we have not actually taken that step yet. We probably shouldn't sneak it in here. I don't see anything in the script that should require a specific Python version, so I think it should be tweaked to be version-independent and just invoke "python".

Ah, no problem. I can fix this. In a quick scan it looked like sphinx was python3, but I see now that's just my install. :)

Quoted 5 lines
Quoted 5 lines
Documentation/Makefile +0 -0
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \
 	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
scripts/jobserver-count new file +0 -0
new file mode 100755
Quoted 8 lines

scripts/jobserver-count line 2

Quoted 6 lines

index 000000000000..ff6ebe6b0194

scripts/jobserver-count new file +2 -0
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later

By license-rules.rst, this should be GPL-2.0+

Whoops, thanks.

Quoted 8 lines
Quoted 8 lines
scripts/jobserver-count new file +8 -0
+# Extract and prepare jobserver file descriptors from envirnoment.
+try:
+	# Fetch the make environment options.
+	flags = os.environ['MAKEFLAGS']
+
+	# Look for "--jobserver=R,W"
+	opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
Quoted 8 lines

scripts/jobserver-count line 31

Quoted 4 lines
scripts/jobserver-count new file +4 -0
+	# Parse out R,W file descriptor numbers and set them nonblocking.
+	fds = opts[0].split("=", 1)[1]
+	reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
+except:

So I have come to really dislike bare "except" clauses; I've seen them hide too many bugs. In this case, perhaps it's justified, but still ... it bugs me ...

Fair enough. I will adjust this (and the later instance).

Quoted 17 lines

scripts/jobserver-count line 43

Quoted 13 lines
scripts/jobserver-count new file +12 -0
+	# Any failures here should result in just using the default
+	# specified parallelism.
+	print(default)
+	sys.exit(0)
+
+# Read out as many jobserver slots as possible.
+jobs = b""
+while True:
+	try:
+		slot = os.read(reader, 1)
+		jobs += slot
+	except:

This one, I think, should be explicit; anything other than EWOULDBLOCK indicates a real problem, right?

Quoted 6 lines

scripts/jobserver-count line 46

Quoted 3 lines
scripts/jobserver-count new file +3 -0
+		break
+# Return all the reserved slots.
+os.write(writer, jobs)

You made writer nonblocking, so it seems plausible that we could leak some slots here, no? Does writer really need to be nonblocking?

Good point. I will fix this too.

Quoted 12 lines

scripts/jobserver-count line 53

Quoted 7 lines
scripts/jobserver-count new file +6 -0
+# If the jobserver was (impossibly) full or communication failed, use default.
+if len(jobs) < 1:
+	print(default)
+
+# Report available slots (with a bump for our caller's reserveration).
+print(len(jobs) + 1)

The last question I have is...why is it that we have to do this complex dance rather than just passing the "-j" option through directly to sphinx? That comes down to the "confusion" mentioned at the top, I assume. It would be good to understand that?

There is no method I have found to discover the -j option's contents (intentionally so, it seems) from within make. :(

Signature · 1 line
Kees Cook
#3 lore raw
Quoted 8 lines · Kees Cook
Quoted 7 lines · Jonathan Corbet
Quoted 3 lines · On Thu, 19 Sep 2019 14:44:37 -0700 Kees Cook <[email protected]> wrote:

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available

Quoted 6 lines
Quoted 2 lines

What sort of confusion might we expect? Or, to channel akpm, "what are the user-visible effects of this bug"?

When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not parallelized. When I run "make htmldocs -j8" with 1.7+ sphinx, it uses all my CPUs instead of 8. :)

To be honest, part of the solution should be to require Sphinx 1.8 or later. Even Debian stable has it. If your distro doesn't have it (really?), using the latest Sphinx in a virtual environment should be a matter of:

$ python3 -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install sphinx sphinx_rtd_theme
(.venv) $ make htmldocs

BR,
Jani.

Quoted 89 lines
Quoted 9 lines

Documentation/Makefile quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) · line 71

Quoted 2 lines
Documentation/Makefile +1 -0
+	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \

This (and the shebang line in the script itself) will cause the docs build to fail on systems lacking Python 3. While we have talked about requiring Python 3 for the docs build, we have not actually taken that step yet. We probably shouldn't sneak it in here. I don't see anything in the script that should require a specific Python version, so I think it should be tweaked to be version-independent and just invoke "python".

Ah, no problem. I can fix this. In a quick scan it looked like sphinx was python3, but I see now that's just my install. :)

Quoted 5 lines
Quoted 5 lines
Documentation/Makefile +0 -0
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \
 	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
scripts/jobserver-count new file +0 -0
new file mode 100755
Quoted 8 lines

scripts/jobserver-count line 2

Quoted 6 lines

index 000000000000..ff6ebe6b0194

scripts/jobserver-count new file +2 -0
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later

By license-rules.rst, this should be GPL-2.0+

Whoops, thanks.

Quoted 8 lines
Quoted 8 lines
scripts/jobserver-count new file +8 -0
+# Extract and prepare jobserver file descriptors from envirnoment.
+try:
+	# Fetch the make environment options.
+	flags = os.environ['MAKEFLAGS']
+
+	# Look for "--jobserver=R,W"
+	opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
Quoted 8 lines

scripts/jobserver-count line 31

Quoted 4 lines
scripts/jobserver-count new file +4 -0
+	# Parse out R,W file descriptor numbers and set them nonblocking.
+	fds = opts[0].split("=", 1)[1]
+	reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
+except:

So I have come to really dislike bare "except" clauses; I've seen them hide too many bugs. In this case, perhaps it's justified, but still ... it bugs me ...

Fair enough. I will adjust this (and the later instance).

Quoted 17 lines

scripts/jobserver-count line 43

Quoted 13 lines
scripts/jobserver-count new file +12 -0
+	# Any failures here should result in just using the default
+	# specified parallelism.
+	print(default)
+	sys.exit(0)
+
+# Read out as many jobserver slots as possible.
+jobs = b""
+while True:
+	try:
+		slot = os.read(reader, 1)
+		jobs += slot
+	except:

This one, I think, should be explicit; anything other than EWOULDBLOCK indicates a real problem, right?

Quoted 6 lines

scripts/jobserver-count line 46

Quoted 3 lines
scripts/jobserver-count new file +3 -0
+		break
+# Return all the reserved slots.
+os.write(writer, jobs)

You made writer nonblocking, so it seems plausible that we could leak some slots here, no? Does writer really need to be nonblocking?

Good point. I will fix this too.

Quoted 12 lines

scripts/jobserver-count line 53

Quoted 7 lines
scripts/jobserver-count new file +6 -0
+# If the jobserver was (impossibly) full or communication failed, use default.
+if len(jobs) < 1:
+	print(default)
+
+# Report available slots (with a bump for our caller's reserveration).
+print(len(jobs) + 1)

The last question I have is...why is it that we have to do this complex dance rather than just passing the "-j" option through directly to sphinx? That comes down to the "confusion" mentioned at the top, I assume. It would be good to understand that?

There is no method I have found to discover the -j option's contents (intentionally so, it seems) from within make. :(

Signature · 1 line
Jani Nikula, Intel Open Source Graphics Center
author
#4 lore raw
Quoted 21 lines · Jani Nikula
Quoted 8 lines · Kees Cook
Quoted 7 lines · Jonathan Corbet
Quoted 3 lines · On Thu, 19 Sep 2019 14:44:37 -0700 Kees Cook <[email protected]> wrote:

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available

Quoted 6 lines
Quoted 2 lines

What sort of confusion might we expect? Or, to channel akpm, "what are the user-visible effects of this bug"?

When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not parallelized. When I run "make htmldocs -j8" with 1.7+ sphinx, it uses all my CPUs instead of 8. :)

To be honest, part of the solution should be to require Sphinx 1.8 or later. Even Debian stable has it. If your distro doesn't have it (really?), using the latest Sphinx in a virtual environment should be a matter of:

Quoted 4 lines
$ python3 -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install sphinx sphinx_rtd_theme
(.venv) $ make htmldocs

I don't mind having sphinx 1.8 (I did, in fact, already update it), but that still doesn't solve the whole problem: my -j argument is being ignored...

-Kees

Quoted 97 lines

BR,
Jani.

Quoted 89 lines
Quoted 9 lines

Documentation/Makefile quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) · line 71

Quoted 2 lines
Documentation/Makefile +1 -0
+	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \

This (and the shebang line in the script itself) will cause the docs build to fail on systems lacking Python 3. While we have talked about requiring Python 3 for the docs build, we have not actually taken that step yet. We probably shouldn't sneak it in here. I don't see anything in the script that should require a specific Python version, so I think it should be tweaked to be version-independent and just invoke "python".

Ah, no problem. I can fix this. In a quick scan it looked like sphinx was python3, but I see now that's just my install. :)

Quoted 5 lines
Quoted 5 lines
Documentation/Makefile +0 -0
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \
 	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
scripts/jobserver-count new file +0 -0
new file mode 100755
Quoted 8 lines

scripts/jobserver-count line 2

Quoted 6 lines

index 000000000000..ff6ebe6b0194

scripts/jobserver-count new file +2 -0
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later

By license-rules.rst, this should be GPL-2.0+

Whoops, thanks.

Quoted 8 lines
Quoted 8 lines
scripts/jobserver-count new file +8 -0
+# Extract and prepare jobserver file descriptors from envirnoment.
+try:
+	# Fetch the make environment options.
+	flags = os.environ['MAKEFLAGS']
+
+	# Look for "--jobserver=R,W"
+	opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
Quoted 8 lines

scripts/jobserver-count line 31

Quoted 4 lines
scripts/jobserver-count new file +4 -0
+	# Parse out R,W file descriptor numbers and set them nonblocking.
+	fds = opts[0].split("=", 1)[1]
+	reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
+except:

So I have come to really dislike bare "except" clauses; I've seen them hide too many bugs. In this case, perhaps it's justified, but still ... it bugs me ...

Fair enough. I will adjust this (and the later instance).

Quoted 17 lines

scripts/jobserver-count line 43

Quoted 13 lines
scripts/jobserver-count new file +12 -0
+	# Any failures here should result in just using the default
+	# specified parallelism.
+	print(default)
+	sys.exit(0)
+
+# Read out as many jobserver slots as possible.
+jobs = b""
+while True:
+	try:
+		slot = os.read(reader, 1)
+		jobs += slot
+	except:

This one, I think, should be explicit; anything other than EWOULDBLOCK indicates a real problem, right?

Quoted 6 lines

scripts/jobserver-count line 46

Quoted 3 lines
scripts/jobserver-count new file +3 -0
+		break
+# Return all the reserved slots.
+os.write(writer, jobs)

You made writer nonblocking, so it seems plausible that we could leak some slots here, no? Does writer really need to be nonblocking?

Good point. I will fix this too.

Quoted 12 lines

scripts/jobserver-count line 53

Quoted 7 lines
scripts/jobserver-count new file +6 -0
+# If the jobserver was (impossibly) full or communication failed, use default.
+if len(jobs) < 1:
+	print(default)
+
+# Report available slots (with a bump for our caller's reserveration).
+print(len(jobs) + 1)

The last question I have is...why is it that we have to do this complex dance rather than just passing the "-j" option through directly to sphinx? That comes down to the "confusion" mentioned at the top, I assume. It would be good to understand that?

There is no method I have found to discover the -j option's contents (intentionally so, it seems) from within make. :(

Signature · 1 line
Jani Nikula, Intel Open Source Graphics Center
Signature · 1 line
Kees Cook
#6 lore raw
In reply to Kees Cook

Em Mon, 23 Sep 2019 15:40:41 -0700
Kees Cook <[email protected]> escreveu:

Quoted 8 lines · Jonathan Corbet
Quoted 7 lines
Quoted 3 lines · On Thu, 19 Sep 2019 14:44:37 -0700 Kees Cook <[email protected]> wrote:

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available

Quoted 5 lines
Quoted 2 lines

What sort of confusion might we expect? Or, to channel akpm, "what are the user-visible effects of this bug"?

When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not parallelized.

Sphinx supports parallel builds for a while. With pre-1.7, you could do something like:

make SPHINXOPTS="-j16" htmldocs

Yet, on my experiences on big machines (tested here with Xeon with 40 and 64 CPU threads), parallel build doesn't actually benefit with values higher than -j5 to -j8, with pre-1.7.

Sphinx 1.7 and higher seem to have improved a lot with "-jauto" (although I didn't time it comparing with -j5 or -j8 on a big server).

Quoted 2 lines

When I run "make htmldocs -j8" with 1.7+ sphinx, it uses all my CPUs instead of 8. :)

This should do the trick:

make SPHINXOPTS="-j8" htmldocs

But yeah, IMHO, the best is if it could honor the Makefile flag:

make -j8 htmldocs

If SPHINXOPTS doesn't contain "-j".

Quoted 11 lines
Quoted 8 lines

Documentation/Makefile quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) · line 71

Quoted 1 line
Documentation/Makefile +1 -0
+	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \  

This (and the shebang line in the script itself) will cause the docs build to fail on systems lacking Python 3. While we have talked about requiring Python 3 for the docs build, we have not actually taken that step yet. We probably shouldn't sneak it in here. I don't see anything in the script that should require a specific Python version, so I think it should be tweaked to be version-independent and just invoke "python".

Ah, no problem. I can fix this. In a quick scan it looked like sphinx was python3, but I see now that's just my install. :)

On Fedora 30, both python2 and python3 versions are available:

python2-sphinx.noarch : Python documentation generator python3-sphinx.noarch : Python documentation generator

However, if one tries to install it without specifying "2" or "3", it defaults to python2 version:

	$ sudo dnf install python-sphinx
	...
	Installing:
	 python2-sphinx                  noarch 1:1.8.4-1.fc30            fedora  1.8 M

AFAIKT, this also applies when distro upgrades takes place: upgrading a python2 sphinx from Fedora 30 to Rawhide will very likely keep the python2 version.

Thanks,
Mauro

#5 lore raw
In reply to Kees Cook
Quoted 22 lines · Kees Cook
Quoted 21 lines · Jani Nikula
Quoted 8 lines · Kees Cook
Quoted 7 lines · Jonathan Corbet
Quoted 3 lines · On Thu, 19 Sep 2019 14:44:37 -0700 Kees Cook <[email protected]> wrote:

While sphinx 1.7 and later supports "-jauto" for parallelism, this effectively ignores the "-j" flag used in the "make" invocation, which may cause confusion for build systems. Instead, extract the available

Quoted 6 lines
Quoted 2 lines

What sort of confusion might we expect? Or, to channel akpm, "what are the user-visible effects of this bug"?

When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not parallelized. When I run "make htmldocs -j8" with 1.7+ sphinx, it uses all my CPUs instead of 8. :)

To be honest, part of the solution should be to require Sphinx 1.8 or later. Even Debian stable has it. If your distro doesn't have it (really?), using the latest Sphinx in a virtual environment should be a matter of:

Quoted 8 lines
Quoted 4 lines
$ python3 -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install sphinx sphinx_rtd_theme
(.venv) $ make htmldocs

I don't mind having sphinx 1.8 (I did, in fact, already update it), but that still doesn't solve the whole problem: my -j argument is being ignored...

I meant, *part* of the solution should be to not have to deal with ancient Sphinx.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Graphics Center