Skip to contentSkip to conversation

docs: Fix the docs build with Sphinx 6.0

PATCH In review Jonathan Corbet 4 participants 8 messages Jan 4, 2023 – Jan 14, 2023

Subsystems (2)
lore
Share & tools
lore

Jonathan Corbet

author maintainer
#0 lore raw

PATCH 1/1 docs: Fix the docs build with Sphinx 6.0

Sphinx 6.0 removed the execfile_() function, which we use as part of the configuration process. They *did* warn us... Just open-code the functionality as is done in Sphinx itself.

Tested (using SPHINX_CONF, since this code is only executed with an alternative config file) on various Sphinx versions from 2.5 through 6.0.

---
 Documentation/sphinx/load_config.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Documentation/sphinx/load_config.py +4 -2
index eeb394b39e2c..8b416bfd75ac 100644
@@ -3,7 +3,7 @@
 
 import os
 import sys
-from sphinx.util.pycompat import execfile_
+from sphinx.util.osutil import fs_encoding
 
 # ------------------------------------------------------------------------------
 def loadConfig(namespace):
@@ -48,7 +48,9 @@ def loadConfig(namespace):
             sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
             config = namespace.copy()
             config['__file__'] = config_file
-            execfile_(config_file, config)
+            with open(config_file, 'rb') as f:
+                code = compile(f.read(), fs_encoding, 'exec')
+                exec(code, config)
             del config['__file__']
             namespace.update(config)
         else:
Signature · 1 line
2.38.1
Tested-by on v1 1/1
#1 lore raw
Quoted 9 lines · Jonathan Corbet

Sphinx 6.0 removed the execfile_() function, which we use as part of the configuration process. They *did* warn us... Just open-code the functionality as is done in Sphinx itself.

Tested (using SPHINX_CONF, since this code is only executed with an alternative config file) on various Sphinx versions from 2.5 through 6.0.

I have tested full builds of documentation with this change with Sphinx versions 1.7.9, 2.4.5, 3.4.3, 4.5.0, 5.3.0, and 6.0.0.

That said, Sphinx 6.0.0 needs much more time and memory than earlier versions.

FYI, I needed to limit parallel slot to 2 (make -j2) on a 16GB machine. If you are lucky, -j3 and -j4 might succeed. -j5 or more ended up in OOM situations for me:

Comparison of elapsed time and maxresident with -j2:

============== ============ ===========
Sphinx version elapsed time maxresident
============== ============ ===========
5.3.0 10:16.81 937660
6.0.0 17:29.07 5292392
============== ============ ===========

Thanks, Akira

Quoted 30 lines
---
 Documentation/sphinx/load_config.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Documentation/sphinx/load_config.py +4 -2
index eeb394b39e2c..8b416bfd75ac 100644
@@ -3,7 +3,7 @@
 
 import os
 import sys
-from sphinx.util.pycompat import execfile_
+from sphinx.util.osutil import fs_encoding
 
 # ------------------------------------------------------------------------------
 def loadConfig(namespace):
@@ -48,7 +48,9 @@ def loadConfig(namespace):
             sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
             config = namespace.copy()
             config['__file__'] = config_file
-            execfile_(config_file, config)
+            with open(config_file, 'rb') as f:
+                code = compile(f.read(), fs_encoding, 'exec')
+                exec(code, config)
             del config['__file__']
             namespace.update(config)
         else:
Signature · 1 line
2.38.1
#2 lore raw

Em Sat, 7 Jan 2023 14:17:24 +0900
Akira Yokosawa <[email protected]> escreveu:

Quoted 23 lines · Jonathan Corbet
Quoted 9 lines

Sphinx 6.0 removed the execfile_() function, which we use as part of the configuration process. They *did* warn us... Just open-code the functionality as is done in Sphinx itself.

Tested (using SPHINX_CONF, since this code is only executed with an alternative config file) on various Sphinx versions from 2.5 through 6.0.

I have tested full builds of documentation with this change with Sphinx versions 1.7.9, 2.4.5, 3.4.3, 4.5.0, 5.3.0, and 6.0.0.

That said, Sphinx 6.0.0 needs much more time and memory than earlier versions.

FYI, I needed to limit parallel slot to 2 (make -j2) on a 16GB machine. If you are lucky, -j3 and -j4 might succeed. -j5 or more ended up in OOM situations for me:

Quoted 8 lines

Comparison of elapsed time and maxresident with -j2:

============== ============ ===========
Sphinx version elapsed time maxresident
============== ============ ===========
5.3.0 10:16.81 937660
6.0.0 17:29.07 5292392
============== ============ ===========

From the changelogs:
https://www.sphinx-doc.org/en/master/changes.html

It seems that 6.1 came with some performance optimizations, in particular:

Cache doctrees in the build environment during the writing phase.

Make all writing phase tasks support parallel execution.

Cache doctrees between the reading and writing phases.

It would be nice if you could also test and check elapsed time there too, as I suspect that 6.0 will have a very short usage, as 6.1 was released just a few days after it.

Regards,
Mauro.

Quoted 18 lines

Thanks, Akira

Quoted 15 lines
---
 Documentation/sphinx/load_config.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Documentation/sphinx/load_config.py +1 -1
index eeb394b39e2c..8b416bfd75ac 100644
@@ -3,7 +3,7 @@
 
 import os
 import sys
-from sphinx.util.pycompat import execfile_
+from sphinx.util.osutil import fs_encoding
 
Quoted 15 lines
Quoted 15 lines
Documentation/sphinx/load_config.py +3 -1
 # ------------------------------------------------------------------------------
 def loadConfig(namespace):
@@ -48,7 +48,9 @@ def loadConfig(namespace):
             sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
             config = namespace.copy()
             config['__file__'] = config_file
-            execfile_(config_file, config)
+            with open(config_file, 'rb') as f:
+                code = compile(f.read(), fs_encoding, 'exec')
+                exec(code, config)
             del config['__file__']
             namespace.update(config)
         else:
Signature · 1 line
2.38.1  

Thanks,
Mauro

#3 lore raw
Quoted 34 lines · Mauro Carvalho Chehab

Em Sat, 7 Jan 2023 14:17:24 +0900
Akira Yokosawa <[email protected]> escreveu:

Quoted 31 lines · Jonathan Corbet
Quoted 9 lines

Sphinx 6.0 removed the execfile_() function, which we use as part of the configuration process. They *did* warn us... Just open-code the functionality as is done in Sphinx itself.

Tested (using SPHINX_CONF, since this code is only executed with an alternative config file) on various Sphinx versions from 2.5 through 6.0.

I have tested full builds of documentation with this change with Sphinx versions 1.7.9, 2.4.5, 3.4.3, 4.5.0, 5.3.0, and 6.0.0.

That said, Sphinx 6.0.0 needs much more time and memory than earlier versions.

FYI, I needed to limit parallel slot to 2 (make -j2) on a 16GB machine. If you are lucky, -j3 and -j4 might succeed. -j5 or more ended up in OOM situations for me:

Comparison of elapsed time and maxresident with -j2:

   ============== ============ ===========
   Sphinx version elapsed time maxresident
   ============== ============ ===========
   5.3.0          10:16.81      937660
   6.0.0          17:29.07     5292392
   ============== ============ ===========

Hi.

I can confirm the regression, I bisected Sphinx revision that caused that and filled an upstream issues:
https://github.com/sphinx-doc/sphinx/issues/11116

Cheers,
Martin

Quoted 59 lines

From the changelogs:
https://www.sphinx-doc.org/en/master/changes.html

It seems that 6.1 came with some performance optimizations, in particular:

Cache doctrees in the build environment during the writing phase.

Make all writing phase tasks support parallel execution.

Cache doctrees between the reading and writing phases.

It would be nice if you could also test and check elapsed time there too, as I suspect that 6.0 will have a very short usage, as 6.1 was released just a few days after it.

Regards,
Mauro.

Quoted 18 lines

Thanks, Akira

Quoted 15 lines
---
  Documentation/sphinx/load_config.py | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
Documentation/sphinx/load_config.py +1 -1
index eeb394b39e2c..8b416bfd75ac 100644
@@ -3,7 +3,7 @@
 
  import os
  import sys
-from sphinx.util.pycompat import execfile_
+from sphinx.util.osutil import fs_encoding
 
Quoted 15 lines
Quoted 15 lines
Documentation/sphinx/load_config.py +3 -1
  # ------------------------------------------------------------------------------
  def loadConfig(namespace):
@@ -48,7 +48,9 @@ def loadConfig(namespace):
              sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
              config = namespace.copy()
              config['__file__'] = config_file
-            execfile_(config_file, config)
+            with open(config_file, 'rb') as f:
+                code = compile(f.read(), fs_encoding, 'exec')
+                exec(code, config)
              del config['__file__']
              namespace.update(config)
          else:
Signature · 1 line
2.38.1

Thanks,
Mauro

#4 lore raw
Quoted 36 lines · Martin Liška
Quoted 34 lines · Mauro Carvalho Chehab

Em Sat, 7 Jan 2023 14:17:24 +0900
Akira Yokosawa <[email protected]> escreveu:

Quoted 31 lines · Jonathan Corbet
Quoted 9 lines

Sphinx 6.0 removed the execfile_() function, which we use as part of the configuration process.  They *did* warn us...  Just open-code the functionality as is done in Sphinx itself.

Tested (using SPHINX_CONF, since this code is only executed with an alternative config file) on various Sphinx versions from 2.5 through 6.0.

I have tested full builds of documentation with this change with Sphinx versions 1.7.9, 2.4.5, 3.4.3, 4.5.0, 5.3.0, and 6.0.0.

That said, Sphinx 6.0.0 needs much more time and memory than earlier versions.

FYI, I needed to limit parallel slot to 2 (make -j2) on a 16GB machine. If you are lucky, -j3 and -j4 might succeed. -j5 or more ended up in OOM situations for me:

Comparison of elapsed time and maxresident with -j2:

   ============== ============ ===========
   Sphinx version elapsed time maxresident
   ============== ============ ===========
   5.3.0          10:16.81      937660
   6.0.0          17:29.07     5292392
   ============== ============ ===========

Quoted 5 lines

Hi.

I can confirm the regression, I bisected Sphinx revision that caused that and filled an upstream issues:
https://github.com/sphinx-doc/sphinx/issues/11116

Thank you Martin for looking into this!

Quoted 19 lines

Cheers,
Martin

Quoted 15 lines

 From the changelogs:
    https://www.sphinx-doc.org/en/master/changes.html

It seems that 6.1 came with some performance optimizations, in particular:

     Cache doctrees in the build environment during the writing phase.

     Make all writing phase tasks support parallel execution.

     Cache doctrees between the reading and writing phases.

It would be nice if you could also test and check elapsed time there too, as I suspect that 6.0 will have a very short usage, as 6.1 was released just a few days after it.

Here is a table comparing 5.3.0, 6.0.1 and 6.1.2 taken on the same machine (with 16GiB mem + 2GiB swap):

====== ===================================

                     elapsed time
         -----------------------------------
 Sphinx     -j1     -j2      -j4      -j6
 ======  ======== ======== ======== ========
  6.1.2  15:11.74 18:06.89 16:39.93      OOM
  6.0.1  15:28.19 17:22.15 16:31.30      OOM
  5.3.0  14:13.04 10:16.81  8:22.37  8:09.74
 ======  ======== ======== ======== ========

Note:
- The -j1 run needs an explicit option given to sphinx-build by:
make SPHINXOPTS="-q -j1" htmldocs
- Once an OOM happens, -j4 runs are likely end up in OOM.

Looks like the non-parallel run is the cheapest option for mitigating the regression.

Thanks, Akira

Quoted 8 lines
Quoted 8 lines

Regards,
Mauro.

Quoted 2 lines

         Thanks, Akira

#5 lore raw
Quoted 8 lines · Akira Yokosawa
Quoted 5 lines · Martin Liška

Hi.

I can confirm the regression, I bisected Sphinx revision that caused that and filled an upstream issues:
https://github.com/sphinx-doc/sphinx/issues/11116

Thank you Martin for looking into this!

Thanks to Martin's inputs on the github issue, Sphinx 6.1.3 has released and the issue is resolved for parallel builds.

However, for non-parallel builds, the memory hog still remains. Again, this is a table comparing 5.3.0, 6.1.2, and 6.1.3.

====== =================================== ===============================
elapsed time maxresident
----------------------------------- -------------------------------
Sphinx -j1 -j2 -j4 -j6 -j1 -j2 -j4 -j6

 ======  ======== ======== ======== ========  ======= ======= ======= =======
  6.1.3  15:03.83 11:31.99  9:35.15  8:49.01  2949056 1059516  978232  967400
  6.1.2  15:11.74 18:06.89 16:39.93      OOM  2961524 5548344 5255372      --
  5.3.0  14:13.04 10:16.81  8:22.37  8:09.74   711532  937660  846016  800340
 ======  ===================================  ===============================

Note:
- The -j1 run needs an explicit option given to sphinx-build:
make SPHINXOPTS="-q -j1" htmldocs

I naively assumed that the memory hog would be resolved all together, but that's not the case.

Martin, could you report the remaining issue to upstream Sphinx?

Thanks, Akira

#6 lore raw
Quoted 4 lines · Akira Yokosawa
Quoted 3 lines · Akira Yokosawa
Quoted 2 lines · Martin Liška

Hi.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I can confirm the regression, I bisected Sphinx revision that caused that and filled an upstream issues:
https://github.com/sphinx-doc/sphinx/issues/11116

Thank you Martin for looking into this!

Thanks to Martin's inputs on the github issue, Sphinx 6.1.3 has released and the issue is resolved for parallel builds.

You're welcome.

Quoted 8 lines

However, for non-parallel builds, the memory hog still remains. Again, this is a table comparing 5.3.0, 6.1.2, and 6.1.3.

====== =================================== ===============================
elapsed time maxresident
----------------------------------- -------------------------------
Sphinx -j1 -j2 -j4 -j6 -j1 -j2 -j4 -j6

Quoted 5 lines
 ======  ======== ======== ======== ========  ======= ======= ======= =======
  6.1.3  15:03.83 11:31.99  9:35.15  8:49.01  2949056 1059516  978232  967400
  6.1.2  15:11.74 18:06.89 16:39.93      OOM  2961524 5548344 5255372      --
  5.3.0  14:13.04 10:16.81  8:22.37  8:09.74   711532  937660  846016  800340
 ======  ===================================  ===============================

I thank you for the nice numbers you provided.

Quoted 7 lines

Note:
- The -j1 run needs an explicit option given to sphinx-build:
make SPHINXOPTS="-q -j1" htmldocs

I naively assumed that the memory hog would be resolved all together, but that's not the case.

Yep, I would expect that same.

Quoted 2 lines

Martin, could you report the remaining issue to upstream Sphinx?

Sure: https://github.com/sphinx-doc/sphinx/issues/11124

Btw. do you have an Github account I can CC?

Cheers,
Martin

Quoted 4 lines

Thanks, Akira

#7 lore raw
Quoted 30 lines · Martin Liška
Quoted 4 lines · Akira Yokosawa
Quoted 3 lines · Akira Yokosawa
Quoted 2 lines · Martin Liška

Hi.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I can confirm the regression, I bisected Sphinx revision that caused that and filled an upstream issues:
https://github.com/sphinx-doc/sphinx/issues/11116

Thank you Martin for looking into this!

Thanks to Martin's inputs on the github issue, Sphinx 6.1.3 has released and the issue is resolved for parallel builds.

You're welcome.

Quoted 8 lines

However, for non-parallel builds, the memory hog still remains. Again, this is a table comparing 5.3.0, 6.1.2, and 6.1.3.

====== =================================== ===============================
elapsed time maxresident
----------------------------------- -------------------------------
Sphinx -j1 -j2 -j4 -j6 -j1 -j2 -j4 -j6

Quoted 5 lines
 ======  ======== ======== ======== ========  ======= ======= ======= =======
  6.1.3  15:03.83 11:31.99  9:35.15  8:49.01  2949056 1059516  978232  967400
  6.1.2  15:11.74 18:06.89 16:39.93      OOM  2961524 5548344 5255372      --
  5.3.0  14:13.04 10:16.81  8:22.37  8:09.74   711532  937660  846016  800340
 ======  ===================================  ===============================
Quoted 1 line

I thank you for the nice numbers you provided.

You are welcome.

Quoted 9 lines
Quoted 8 lines

Note:
- The -j1 run needs an explicit option given to sphinx-build:
make SPHINXOPTS="-q -j1" htmldocs

I naively assumed that the memory hog would be resolved all together, but that's not the case.

Quoted 6 lines

Yep, I would expect that same.

Quoted 2 lines

Martin, could you report the remaining issue to upstream Sphinx?

Sure: https://github.com/sphinx-doc/sphinx/issues/11124

Thanks!

Quoted 2 lines

Btw. do you have an Github account I can CC?

I have reacted with an emoji and subscribed to the issue.

Thanks, Akira

Quoted 9 lines

Cheers,
Martin

Quoted 4 lines

Thanks, Akira