Don't create Python bytecode when building the kernel
Subsystems (6)
v3 cover Don't create Python bytecode when building the kernel
As reported by Andy, the Kernel build system runs kernel-doc script for DRM, when W=1. Due to Python's normal behavior, its JIT compiler will create a bytecode and store it under scripts/lib/*/__pycache__.
As one may be using O= and even having the sources on a read-only mount point, disable its creation during build time.
This only solves half of the issue though, as one may be manually running the script by hand, without asking Python to not store any bytecode. This should be OK, but afterwards, git status will list the __pycache__ as not committed. To prevent that, add *.pyc to .gitignore.
---
v3:
- changed the order of PYTHONDONTBYTECODE;
- patched also scripts/Makefile
Mauro Carvalho Chehab (2):
scripts/kernel-doc.py: don't create *.pyc files
.gitignore: ignore Python compiled bytecode
.gitignore | 1 + drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 6 files changed, 6 insertions(+), 5 deletions(-)
Signature · 1 line
2.49.0
PATCH v3 1/2 scripts/kernel-doc.py: don't create *.pyc files
As reported by Andy, kernel-doc.py is creating a __pycache__ directory at build time.
Disable creation of __pycache__ for the libraries used by kernel-doc.py, when excecuted via the build system or via scripts/find-unused-docs.sh.
--- drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
drivers/gpu/drm/Makefile +1 -1
@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ touch $@ $(obj)/%.hdrtest: $(src)/%.h FORCE
drivers/gpu/drm/i915/Makefile +1 -1
@@ -408,7 +408,7 @@ obj-$(CONFIG_DRM_I915_GVT_KVMGT) += kvmgt.o # # Enable locally for CONFIG_DRM_I915_WERROR=y. See also scripts/Makefile.build ifdef CONFIG_DRM_I915_WERROR - cmd_checkdoc = $(srctree)/scripts/kernel-doc -none -Werror $< + cmd_checkdoc = PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none -Werror $< endif # header test
include/drm/Makefile +1 -1
@@ -11,7 +11,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ touch $@ $(obj)/%.hdrtest: $(src)/%.h FORCE
scripts/Makefile.build +1 -1
@@ -83,7 +83,7 @@ else ifeq ($(KBUILD_CHECKSRC),2) endif ifneq ($(KBUILD_EXTRA_WARN),) - cmd_checkdoc = $(srctree)/scripts/kernel-doc -none $(KDOCFLAGS) \ + cmd_checkdoc = PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(KDOCFLAGS) \ $(if $(findstring 2, $(KBUILD_EXTRA_WARN)), -Wall) \ $< endif
scripts/find-unused-docs.sh +1 -1
@@ -54,7 +54,7 @@ for file in `find $1 -name '*.c'`; do if [[ ${FILES_INCLUDED[$file]+_} ]]; then continue; fi - str=$(scripts/kernel-doc -export "$file" 2>/dev/null) + str=$(PYTHONDONTWRITEBYTECODE=1 scripts/kernel-doc -export "$file" 2>/dev/null) if [[ -n "$str" ]]; then echo "$file" fi
Signature · 1 line
2.49.0
Jani Nikula
maintainerRe: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Quoted 23 lines · Mauro Carvalho Chehab
As reported by Andy,
kernel-doc.pyis creating a__pycache__directory at build time.Disable creation of
__pycache__for the libraries used bykernel-doc.py, when excecuted via the build system or viascripts/find-unused-docs.sh.Reported-by Andy Shevchenko <[email protected]>Signed-off-by Mauro Carvalho Chehab <[email protected]>--- drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)drivers/gpu/drm/Makefile +0
-0
drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239
Quoted 5 lines
drivers/gpu/drm/Makefile +1
-1@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
KERNELDOC is not set here.
/bin/sh: 1: -none: not found
Quoted 55 lines
drivers/gpu/drm/Makefile +0
-0touch $@ $(obj)/%.hdrtest: $(src)/%.h FORCEdrivers/gpu/drm/i915/Makefile +1
-1@@ -408,7 +408,7 @@ obj-$(CONFIG_DRM_I915_GVT_KVMGT) += kvmgt.o # # Enable locally for CONFIG_DRM_I915_WERROR=y. See also scripts/Makefile.build ifdef CONFIG_DRM_I915_WERROR - cmd_checkdoc = $(srctree)/scripts/kernel-doc -none -Werror $< + cmd_checkdoc = PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none -Werror $< endif # header testinclude/drm/Makefile +1
-1@@ -11,7 +11,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ touch $@ $(obj)/%.hdrtest: $(src)/%.h FORCEscripts/Makefile.build +1
-1@@ -83,7 +83,7 @@ else ifeq ($(KBUILD_CHECKSRC),2) endif ifneq ($(KBUILD_EXTRA_WARN),) - cmd_checkdoc = $(srctree)/scripts/kernel-doc -none $(KDOCFLAGS) \ + cmd_checkdoc = PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(KDOCFLAGS) \ $(if $(findstring 2, $(KBUILD_EXTRA_WARN)), -Wall) \ $< endifscripts/find-unused-docs.sh +1
-1@@ -54,7 +54,7 @@ for file in `find $1 -name '*.c'`; do if [[ ${FILES_INCLUDED[$file]+_} ]]; then continue; fi - str=$(scripts/kernel-doc -export "$file" 2>/dev/null) + str=$(PYTHONDONTWRITEBYTECODE=1 scripts/kernel-doc -export "$file" 2>/dev/null) if [[ -n "$str" ]]; then echo "$file" fi
Signature · 1 line
Jani Nikula, Intel
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Em Wed, 16 Apr 2025 11:34:16 +0300
Jani Nikula <[email protected]> escreveu:
Quoted 33 lines · Mauro Carvalho Chehab
Quoted 23 lines
As reported by Andy,
kernel-doc.pyis creating a__pycache__directory at build time.Disable creation of
__pycache__for the libraries used bykernel-doc.py, when excecuted via the build system or viascripts/find-unused-docs.sh.Reported-by Andy Shevchenko <[email protected]>Signed-off-by Mauro Carvalho Chehab <[email protected]>--- drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)drivers/gpu/drm/Makefile +0
-0drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239
Quoted 5 lines
drivers/gpu/drm/Makefile +1
-1@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \KERNELDOC is not set here.
Quoted 1 line
/bin/sh: 1: -none: not found
Weird. This is set on Documentation/Makefile:
$ grep KERNELDOC Documentation/Makefile KERNELDOC = $(srctree)/scripts/kernel-doc.py ...
drivers/gpu/drm/Makefile should be able to see this variable there...
Quoted 57 lines
Quoted 56 lines
drivers/gpu/drm/Makefile +0
-0touch $@ $(obj)/%.hdrtest: $(src)/%.h FORCEdrivers/gpu/drm/i915/Makefile +1
-1@@ -408,7 +408,7 @@ obj-$(CONFIG_DRM_I915_GVT_KVMGT) += kvmgt.o # # Enable locally for CONFIG_DRM_I915_WERROR=y. See also scripts/Makefile.build ifdef CONFIG_DRM_I915_WERROR - cmd_checkdoc = $(srctree)/scripts/kernel-doc -none -Werror $< + cmd_checkdoc = PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none -Werror $< endif # header testinclude/drm/Makefile +1
-1@@ -11,7 +11,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ touch $@ $(obj)/%.hdrtest: $(src)/%.h FORCEscripts/Makefile.build +1
-1@@ -83,7 +83,7 @@ else ifeq ($(KBUILD_CHECKSRC),2) endif ifneq ($(KBUILD_EXTRA_WARN),) - cmd_checkdoc = $(srctree)/scripts/kernel-doc -none $(KDOCFLAGS) \ + cmd_checkdoc = PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(KDOCFLAGS) \ $(if $(findstring 2, $(KBUILD_EXTRA_WARN)), -Wall) \ $< endifscripts/find-unused-docs.sh +1
-1@@ -54,7 +54,7 @@ for file in `find $1 -name '*.c'`; do if [[ ${FILES_INCLUDED[$file]+_} ]]; then continue; fi - str=$(scripts/kernel-doc -export "$file" 2>/dev/null) + str=$(PYTHONDONTWRITEBYTECODE=1 scripts/kernel-doc -export "$file" 2>/dev/null) if [[ -n "$str" ]]; then echo "$file" fi
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Quoted 3 lines · Mauro Carvalho Chehab
Em Wed, 16 Apr 2025 11:34:16 +0300
Jani Nikula <[email protected]> escreveu:Quoted 1 line · Mauro Carvalho Chehab
...
Quoted 11 lines
Quoted 10 lines
drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239
Quoted 5 lines
quiet_cmd_hdrtest= HDRTEST $(patsubst %.hdrtest,%.h,$@)cmd_hdrtest = \drivers/gpu/drm/Makefile +1
-1$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \KERNELDOC is not set here.
/bin/sh: 1: -none: not found
Quoted 7 lines
Weird. This is set on
Documentation/Makefile:$ grep KERNELDOC Documentation/Makefile KERNELDOC = $(srctree)/scripts/kernel-doc.py ...
drivers/gpu/drm/Makefileshould be able to see this variable there...
Does it work in the case of make drivers/gpu/drm ? This is probably how Jani tried it.
Quoted 1 line
Quoted 1 line
Quoted 1 line
touch $@
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
In reply to Mauro Carvalho ChehabEm Wed, 16 Apr 2025 17:19:17 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:
Quoted 38 lines
Em Wed, 16 Apr 2025 11:34:16 +0300
Jani Nikula <[email protected]> escreveu:Quoted 33 lines · Mauro Carvalho Chehab
Quoted 23 lines
As reported by Andy,
kernel-doc.pyis creating a__pycache__directory at build time.Disable creation of
__pycache__for the libraries used bykernel-doc.py, when excecuted via the build system or viascripts/find-unused-docs.sh.Reported-by Andy Shevchenko <[email protected]>Signed-off-by Mauro Carvalho Chehab <[email protected]>--- drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)drivers/gpu/drm/Makefile +0
-0drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239
Quoted 5 lines
drivers/gpu/drm/Makefile +1
-1@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \KERNELDOC is not set here.
Quoted 1 line
/bin/sh: 1: -none: not found
Quoted 7 lines
Weird. This is set on
Documentation/Makefile:$ grep KERNELDOC Documentation/Makefile KERNELDOC = $(srctree)/scripts/kernel-doc.py ...
drivers/gpu/drm/Makefileshould be able to see this variable there...
I suspect that the building system tries to confine variables to sub-directories, so maybe one solution would be to move it to the main makefile.
could you please check if this patch solves the issue?
[PATCH] Makefile: move KERNELDOC macro to the main Makefile
As kernel-doc script is used not only on Documentation, but also on scripts and drivers/drm Makefiles, move it to the main makefile, as otherwise sub-makefiles may not have it.
Documentation/Makefile +0 -1
@@ -60,7 +60,6 @@ endif #HAVE_LATEXMK # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter -KERNELDOC = $(srctree)/scripts/kernel-doc.py KERNELDOC_CONF = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC) ALLSPHINXOPTS = $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) ifneq ($(wildcard $(srctree)/.config),)
Makefile +2 -0
@@ -458,6 +458,8 @@ endif HOSTRUSTC = rustc HOSTPKG_CONFIG = pkg-config +KERNELDOC = $(srctree)/scripts/kernel-doc.py + KBUILD_USERHOSTCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \ -O2 -fomit-frame-pointer -std=gnu11 KBUILD_USERCFLAGS := $(KBUILD_USERHOSTCFLAGS) $(USERCFLAGS)
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Quoted 5 lines · Mauro Carvalho Chehab
Em Wed, 16 Apr 2025 17:19:17 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:Quoted 3 lines
Em Wed, 16 Apr 2025 11:34:16 +0300
Jani Nikula <[email protected]> escreveu:Quoted 1 line · Mauro Carvalho Chehab
...
Quoted 38 lines
Quoted 11 lines
Quoted 10 lines
Review
Quoted 5 lines
quiet_cmd_hdrtest= HDRTEST $(patsubst %.hdrtest,%.h,$@)cmd_hdrtest = \(quoted diff) +1
-1$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \KERNELDOC is not set here.
/bin/sh: 1: -none: not foundQuoted 7 lines
Weird. This is set on
Documentation/Makefile:$ grep KERNELDOC Documentation/Makefile KERNELDOC = $(srctree)/scripts/kernel-doc.py ...
drivers/gpu/drm/Makefileshould be able to see this variable there...I suspect that the building system tries to confine variables to sub-directories, so maybe one solution would be to move it to the main makefile.
could you please check if this patch solves the issue?
[PATCH]
Makefile: move KERNELDOC macro to the mainMakefileAs kernel-doc script is used not only on Documentation, but also on scripts and
drivers/drmMakefiles, move it to the main makefile, as otherwise sub-makefiles may not have it.Signed-off-by Mauro Carvalho Chehab <[email protected]>Documentation/Makefile +0
-0
Documentation/Makefile endif #HAVE_LATEXMK · line 63
Quoted 5 lines
Documentation/Makefile +0
-1@@ -60,7 +60,6 @@ endif #HAVE_LATEXMK # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter -KERNELDOC = $(srctree)/scripts/kernel-doc.py KERNELDOC_CONF = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
In this case the _CONF makes sense to move together as they are coupled semantically.
Quoted 15 lines
Documentation/Makefile +0
-0ALLSPHINXOPTS = $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) ifneq ($(wildcard $(srctree)/.config),)Makefile +2
-0@@ -458,6 +458,8 @@ endif HOSTRUSTC = rustc HOSTPKG_CONFIG = pkg-config +KERNELDOC = $(srctree)/scripts/kernel-doc.py + KBUILD_USERHOSTCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \ -O2 -fomit-frame-pointer -std=gnu11 KBUILD_USERCFLAGS := $(KBUILD_USERHOSTCFLAGS) $(USERCFLAGS)
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Em Wed, 16 Apr 2025 12:36:06 +0300
Andy Shevchenko <[email protected]> escreveu:
Quoted 42 lines · Mauro Carvalho Chehab
Quoted 5 lines
Em Wed, 16 Apr 2025 17:19:17 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:Quoted 3 lines
Em Wed, 16 Apr 2025 11:34:16 +0300
Jani Nikula <[email protected]> escreveu:Quoted 1 line · Mauro Carvalho Chehab
...
Quoted 33 lines
Quoted 11 lines
Quoted 10 lines
Review
Quoted 5 lines
quiet_cmd_hdrtest= HDRTEST $(patsubst %.hdrtest,%.h,$@)cmd_hdrtest = \(quoted diff) +1
-1$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \KERNELDOC is not set here.
/bin/sh: 1: -none: not foundQuoted 7 lines
Weird. This is set on
Documentation/Makefile:$ grep KERNELDOC Documentation/Makefile KERNELDOC = $(srctree)/scripts/kernel-doc.py ...
drivers/gpu/drm/Makefileshould be able to see this variable there...I suspect that the building system tries to confine variables to sub-directories, so maybe one solution would be to move it to the main makefile.
could you please check if this patch solves the issue?
[PATCH]
Makefile: move KERNELDOC macro to the mainMakefileAs kernel-doc script is used not only on Documentation, but also on scripts and
drivers/drmMakefiles, move it to the main makefile, as otherwise sub-makefiles may not have it.Signed-off-by Mauro Carvalho Chehab <[email protected]>
Quoted 13 lines
Documentation/Makefile endif #HAVE_LATEXMK · line 63
Quoted 10 lines
Documentation/Makefile +0
-1@@ -60,7 +60,6 @@ endif #HAVE_LATEXMK # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter -KERNELDOC = $(srctree)/scripts/kernel-doc.py KERNELDOC_CONF = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)In this case the
_CONFmakes sense to move together as they are coupled semantically.
In a matter of fact, it doesn't...
Quoted 1 line
Documentation/Makefile endif #HAVE_LATEXMK · line 64
Quoted 1 line
Documentation/Makefile +0
-0ALLSPHINXOPTS = $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
This is just part of ALLSPHINXOPTS, where it places two definitions to be used by the Sphinx kerneldoc extension. I need to double-check it, but I suspect that this is not even used there anymore. If it is still used, it can be cleaned up after we remove the Perl version.
So, I prefer to keep this (perhaps with a different name) at the documentation makefile.
Regards,
Mauro
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
In reply to Mauro Carvalho ChehabEm Wed, 16 Apr 2025 17:29:01 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:
Quoted 47 lines
Em Wed, 16 Apr 2025 17:19:17 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:Quoted 44 lines
Em Wed, 16 Apr 2025 11:34:16 +0300
Jani Nikula <[email protected]> escreveu:Quoted 33 lines · Mauro Carvalho Chehab
Quoted 23 lines
As reported by Andy,
kernel-doc.pyis creating a__pycache__directory at build time.Disable creation of
__pycache__for the libraries used bykernel-doc.py, when excecuted via the build system or viascripts/find-unused-docs.sh.Reported-by Andy Shevchenko <[email protected]>Signed-off-by Mauro Carvalho Chehab <[email protected]>--- drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)drivers/gpu/drm/Makefile +0
-0drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239
Quoted 5 lines
drivers/gpu/drm/Makefile +1
-1@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) cmd_hdrtest = \ $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ - $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ + PYTHONDONTWRITEBYTECODE=1 $(KERNELDOC) -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \KERNELDOC is not set here.
Quoted 1 line
/bin/sh: 1: -none: not foundWeird. This is set on
Documentation/Makefile:$ grep KERNELDOC Documentation/Makefile KERNELDOC = $(srctree)/scripts/kernel-doc.py ...
Quoted 7 lines
Quoted 1 line
drivers/gpu/drm/Makefileshould be able to see this variable there...I suspect that the building system tries to confine variables to sub-directories, so maybe one solution would be to move it to the main makefile.
could you please check if this patch solves the issue?
Answering myself: it doesn't.
Heh, trying to quickly write a patch before calling it a day is usually not a good idea ;-)
I'll send a fix tomorrow.
Regards,
Mauro
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Quoted 2 lines · Mauro Carvalho Chehab
Em Wed, 16 Apr 2025 17:29:01 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:
...
Quoted 4 lines
Heh, trying to quickly write a patch before calling it a day is usually not a good idea ;-)
I'll send a fix tomorrow.
Take your time, we still have a couple or so weeks to address this.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
Hi Andy,
Em Wed, 16 Apr 2025 12:41:29 +0300
Andy Shevchenko <[email protected]> escreveu:
Quoted 12 lines · Mauro Carvalho Chehab
Quoted 2 lines
Em Wed, 16 Apr 2025 17:29:01 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:...
Quoted 4 lines
Heh, trying to quickly write a patch before calling it a day is usually not a good idea ;-)
I'll send a fix tomorrow.
Take your time, we still have a couple or so weeks to address this.
Sent a v3. Please check.
Regards,
Mauro
Re: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
In reply to Mauro Carvalho ChehabQuoted 2 lines · Mauro Carvalho Chehab
Em Wed, 16 Apr 2025 17:19:17 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:
...
Quoted 1 line
could you please check if this patch solves the issue?
Doesn't work for me.
Signature · 2 lines
With Best Regards, Andy Shevchenko
kernel test robot
build botRe: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
In reply to Mauro Carvalho Chehabkernel test robot noticed the following build errors:
Hi Mauro,
kernel test robot noticed the following build errors:
[auto build test ERROR on lwn/docs-next]
[also build test ERROR on drm-i915/for-linux-next drm-i915/for-linux-next-fixes linus/master v6.15-rc2 next-20250416] [cannot apply to masahiroy-kbuild/for-next masahiroy-kbuild/fixes]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Mauro-Carvalho-Chehab/scripts-kernel-doc-py-don-t-create-pyc-files/20250416-155336 base: git://git.lwn.net/linux.git docs-next patch link: https://lore.kernel.org/r/4ad5eb8d4b819997c1615d2401581c22a32bb2c1.1744789777.git.mchehab%2Bhuawei%40kernel.org patch subject: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files config: csky-randconfig-001-20250417 (https://download.01.org/0day-ci/archive/20250417/[email protected]/config) compiler: csky-linux-gcc (GCC) 13.3.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250417/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> /bin/sh: 1: -none: not found
make[3]: *** [scripts/Makefile.build:203: scripts/mod/empty.o] Error 127 shuffle=2754938148 make[3]: *** Deleting file 'scripts/mod/empty.o'
make[3]: Target 'scripts/mod/' not remade because of errors.
make[2]: *** [Makefile:1276: prepare0] Error 2 shuffle=2754938148
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2 shuffle=2754938148
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2 shuffle=2754938148
make: Target 'prepare' not remade because of errors.
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files
In reply to Mauro Carvalho Chehabkernel test robot noticed the following build errors:
Hi Mauro,
kernel test robot noticed the following build errors:
[auto build test ERROR on lwn/docs-next]
[also build test ERROR on drm-i915/for-linux-next drm-i915/for-linux-next-fixes linus/master v6.15-rc2 next-20250417] [cannot apply to masahiroy-kbuild/for-next masahiroy-kbuild/fixes]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Mauro-Carvalho-Chehab/scripts-kernel-doc-py-don-t-create-pyc-files/20250416-155336 base: git://git.lwn.net/linux.git docs-next patch link: https://lore.kernel.org/r/4ad5eb8d4b819997c1615d2401581c22a32bb2c1.1744789777.git.mchehab%2Bhuawei%40kernel.org patch subject: [PATCH v3 1/2] scripts/kernel-doc.py: don't create *.pyc files config: x86_64-rhel-9.4-rust (https://download.01.org/0day-ci/archive/20250419/[email protected]/config) compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff) rustc: rustc 1.78.0 (9b00956e5 2024-04-29) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250419/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> /bin/bash: line 1: -none: command not found
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v3 2/2 .gitignore: ignore Python compiled bytecode
In reply to Mauro Carvalho ChehabWhile the building system doesn't create any Python JIT bytecode, if one manually runs kernel-doc.py or get_abi.py, Python will, by default, create a bytecode and store it under scripts/lib/*.
This is normal, and not controlled by the Kernel itself. So, add *.pyc as an extension to be ignored.
--- .gitignore | 1 + 1 file changed, 1 insertion(+)
.gitignore +1 -0
@@ -39,6 +39,7 @@ *.mod.c *.o *.o.* +*.pyc *.patch *.rmeta *.rpm
Signature · 1 line
2.49.0
Re: [PATCH v3 2/2] .gitignore: ignore Python compiled bytecode
Quoted 6 lines · Mauro Carvalho Chehab
While the building system doesn't create any Python JIT bytecode, if one manually runs
kernel-doc.pyorget_abi.py, Python will, by default, create a bytecode and store it underscripts/lib/*.This is normal, and not controlled by the Kernel itself. So, add *.pyc as an extension to be ignored.
...
Quoted 7 lines
*.mod.c
*.o
*.o.*
+*.pyc
*.patch
*.rmeta
*.rpm
Same comment as per v2. This list seems ordered.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 7 lines · Mauro Carvalho Chehab
As reported by Andy, the Kernel build system runs kernel-doc script for DRM, when W=1. Due to Python's normal behavior, its JIT compiler will create a bytecode and store it under
scripts/lib/*/__pycache__.As one may be using O= and even having the sources on a read-only mount point, disable its creation during build time.
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
Quoted 28 lines
This only solves half of the issue though, as one may be manually running the script by hand, without asking Python to not store any bytecode. This should be OK, but afterwards, git status will list the
__pycache__as not committed. To prevent that, add *.pyc to .gitignore.---
v3:
- changed the order of PYTHONDONTBYTECODE;
- patched alsoscripts/MakefileMauro Carvalho Chehab (2):
scripts/kernel-doc.py: don't create *.pyc files
.gitignore: ignore Python compiled bytecode.gitignore | 1 + drivers/gpu/drm/Makefile | 2 +- drivers/gpu/drm/i915/Makefile | 2 +- include/drm/Makefile | 2 +- scripts/Makefile.build | 2 +- scripts/find-unused-docs.sh | 2 +- 6 files changed, 6 insertions(+), 5 deletions(-)Signature · 1 line
2.49.0
Signature · 2 lines
With best wishes Dmitry
Jonathan Corbet
maintainerQuoted 11 lines · Dmitry Baryshkov
Quoted 7 lines · Mauro Carvalho Chehab
As reported by Andy, the Kernel build system runs kernel-doc script for DRM, when W=1. Due to Python's normal behavior, its JIT compiler will create a bytecode and store it under
scripts/lib/*/__pycache__.As one may be using O= and even having the sources on a read-only mount point, disable its creation during build time.
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
Thanks,
jon
Quoted 13 lines · Jonathan Corbet
Quoted 11 lines · Dmitry Baryshkov
Quoted 7 lines · Mauro Carvalho Chehab
As reported by Andy, the Kernel build system runs kernel-doc script for DRM, when W=1. Due to Python's normal behavior, its JIT compiler will create a bytecode and store it under
scripts/lib/*/__pycache__.As one may be using O= and even having the sources on a read-only mount point, disable its creation during build time.
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
Quoted 7 lines
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Hi Andy,
Quoted 2 lines · Andy Shevchenko
Quoted 1 line · Jonathan Corbet
Dmitry Baryshkov <[email protected]> writes:
[...]
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
Quoted 8 lines
Quoted 2 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
But, PYTHONPYCACHEPREFIX can be set as an environment variable.
For example, try:
export PYTHONPYCACHEPREFIX="$HOME/.cache/__pycache__"
Wouldn't it be good enough for you?
Regards,
Akira
Quoted 3 lines · Akira Yokosawa
Quoted 2 lines · Andy Shevchenko
Quoted 1 line · Jonathan Corbet
Dmitry Baryshkov <[email protected]> writes:
[...]
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
Quoted 8 lines
Quoted 2 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
Quoted 7 lines
But, PYTHONPYCACHEPREFIX can be set as an environment variable.
For example, try:
export PYTHONPYCACHEPREFIX="$HOME/.cache/__pycache__"
Wouldn't it be good enough for you?
Of course not. We have _many_ scripts in python in kernel and having a cache there for _all_ of them is simply WRONG. You never know what clashes can be there with two complicated enough scripts which may have same module names, etc.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 28 lines · Andy Shevchenko
Quoted 3 lines · Akira Yokosawa
Quoted 2 lines · Andy Shevchenko
Quoted 1 line · Jonathan Corbet
Dmitry Baryshkov <[email protected]> writes:
[...]
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
Quoted 8 lines
Quoted 2 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
But, PYTHONPYCACHEPREFIX can be set as an environment variable.
For example, try:
Quoted 9 lines
Quoted 3 lines
export PYTHONPYCACHEPREFIX="$HOME/.cache/__pycache__"
Wouldn't it be good enough for you?
Of course not. We have _many_ scripts in python in kernel and having a cache there for _all_ of them is simply WRONG. You never know what clashes can be there with two complicated enough scripts which may have same module names, etc.
Interesting...
I'm suspecting you replied without having tried the setting...
FYI, this is an excerpt from list of .pyc files under __pycache__ after building defconfig kernel and "make htmldocs"; and running
$ find . -name *.pyc" -print" under ~/.cache/__pycache__ --------------------------------------------------------------------- ./home/.../git/linux/scripts/lib/kdoc/kdoc_files.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_parser.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_re.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_output.cpython-312.pyc [...] ./usr/lib/python3.12/xml/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/parsers/expat.cpython-312.pyc ./usr/lib/python3.12/xml/parsers/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/etree/ElementPath.cpython-312.pyc ./usr/lib/python3.12/xml/etree/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/etree/cElementTree.cpython-312.pyc ./usr/lib/python3.12/xml/etree/ElementTree.cpython-312.pyc ./usr/lib/python3.12/mimetypes.cpython-312.pyc [...] ./usr/lib/python3/dist-packages/sphinx/deprecation.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/highlighting.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/ast.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/__init__.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/parser.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/config.cpython-312.pyc [...] ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/deprecation.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/highlighting.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/ast.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/__init__.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/parser.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/config.cpython-312.pyc [...] -------------------------------------------------------------------------
As you see, each of them are stored at a path corresponding to its original .py file. The final part of the excerpt came from me running in-development Sphinx in a python venv with the same PYTHONPYCACHEPREFIX setting.
I don't see any possibility of clashes as you mentioned above, quoting again:
Quoted 4 lines
We have _many_ scripts in python in kernel and having a cache
there for _all_ of them is simply WRONG. You never know what clashes can be there with two complicated enough scripts which may have same module names, etc.
Or my imagination might be too limited to see your point ...
Regards,
Akira
Em Thu, 24 Apr 2025 11:07:05 +0900
Akira Yokosawa <[email protected]> escreveu:
Quoted 78 lines · Andy Shevchenko
Quoted 28 lines · Akira Yokosawa
Quoted 3 lines · Andy Shevchenko
Quoted 2 lines · Jonathan Corbet
Quoted 1 line
Dmitry Baryshkov <[email protected]> writes:
[...]
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
Quoted 8 lines
Quoted 2 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
But, PYTHONPYCACHEPREFIX can be set as an environment variable.
For example, try:
Quoted 9 lines
Quoted 3 lines
export PYTHONPYCACHEPREFIX="$HOME/.cache/__pycache__"
Wouldn't it be good enough for you?
Of course not. We have _many_ scripts in python in kernel and having a cache there for _all_ of them is simply WRONG. You never know what clashes can be there with two complicated enough scripts which may have same module names, etc.
Interesting...
I'm suspecting you replied without having tried the setting...
FYI, this is an excerpt from list of .pyc files under
__pycache__after building defconfig kernel and "make htmldocs"; and running$ find . -name *.pyc" -print" under ~/.cache/__pycache__ --------------------------------------------------------------------- ./home/.../git/linux/scripts/lib/kdoc/kdoc_files.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_parser.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_re.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_output.cpython-312.pyc [...] ./usr/lib/python3.12/xml/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/parsers/expat.cpython-312.pyc ./usr/lib/python3.12/xml/parsers/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/etree/ElementPath.cpython-312.pyc ./usr/lib/python3.12/xml/etree/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/etree/cElementTree.cpython-312.pyc ./usr/lib/python3.12/xml/etree/ElementTree.cpython-312.pyc ./usr/lib/python3.12/mimetypes.cpython-312.pyc [...] ./usr/lib/python3/dist-packages/sphinx/deprecation.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/highlighting.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/ast.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/__init__.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/parser.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/config.cpython-312.pyc [...] ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/deprecation.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/highlighting.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/ast.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/__init__.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/parser.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/config.cpython-312.pyc [...] -------------------------------------------------------------------------
Quoted 5 lines
As you see, each of them are stored at a path corresponding to its original .py file. The final part of the excerpt came from me running in-development Sphinx in a python venv with the same PYTHONPYCACHEPREFIX setting.
I don't see any possibility of clashes as you mentioned above,
I didn't test it, but it sounds good enough for me. Could you please send us a patch on the top of:
https://lore.kernel.org/linux-doc/[email protected]/T/#t
for us to test it? The idea is to set PYTHONPYCACHEPREFIX only when O= is used.
Regards,
Mauro
Em Thu, 24 Apr 2025 15:26:48 +0800
Mauro Carvalho Chehab <[email protected]> escreveu:
Quoted 87 lines
Em Thu, 24 Apr 2025 11:07:05 +0900
Akira Yokosawa <[email protected]> escreveu:Quoted 78 lines · Andy Shevchenko
Quoted 28 lines · Akira Yokosawa
Quoted 3 lines · Andy Shevchenko
Quoted 2 lines · Jonathan Corbet
Quoted 1 line
Dmitry Baryshkov <[email protected]> writes:
[...]
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
Quoted 8 lines
Quoted 2 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
But, PYTHONPYCACHEPREFIX can be set as an environment variable.
For example, try:
Quoted 9 lines
Quoted 3 lines
export PYTHONPYCACHEPREFIX="$HOME/.cache/__pycache__"
Wouldn't it be good enough for you?
Of course not. We have _many_ scripts in python in kernel and having a cache there for _all_ of them is simply WRONG. You never know what clashes can be there with two complicated enough scripts which may have same module names, etc.
Interesting...
I'm suspecting you replied without having tried the setting...
FYI, this is an excerpt from list of .pyc files under
__pycache__after building defconfig kernel and "make htmldocs"; and running$ find . -name *.pyc" -print" under ~/.cache/__pycache__ --------------------------------------------------------------------- ./home/.../git/linux/scripts/lib/kdoc/kdoc_files.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_parser.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_re.cpython-312.pyc ./home/.../git/linux/scripts/lib/kdoc/kdoc_output.cpython-312.pyc [...] ./usr/lib/python3.12/xml/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/parsers/expat.cpython-312.pyc ./usr/lib/python3.12/xml/parsers/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/etree/ElementPath.cpython-312.pyc ./usr/lib/python3.12/xml/etree/__init__.cpython-312.pyc ./usr/lib/python3.12/xml/etree/cElementTree.cpython-312.pyc ./usr/lib/python3.12/xml/etree/ElementTree.cpython-312.pyc ./usr/lib/python3.12/mimetypes.cpython-312.pyc [...] ./usr/lib/python3/dist-packages/sphinx/deprecation.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/highlighting.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/ast.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/__init__.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/pycode/parser.cpython-312.pyc ./usr/lib/python3/dist-packages/sphinx/config.cpython-312.pyc [...] ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/deprecation.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/highlighting.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/ast.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/__init__.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/pycode/parser.cpython-312.pyc ./home/.../sphinx-WIP/lib/python3.12/site-packages/sphinx/config.cpython-312.pyc [...] -------------------------------------------------------------------------Quoted 5 lines
As you see, each of them are stored at a path corresponding to its original .py file. The final part of the excerpt came from me running in-development Sphinx in a python venv with the same PYTHONPYCACHEPREFIX setting.
I don't see any possibility of clashes as you mentioned above,
Quoted 7 lines
I didn't test it, but it sounds good enough for me. Could you please send us a patch on the top of:
https://lore.kernel.org/linux-doc/[email protected]/T/#tfor us to test it? The idea is to set PYTHONPYCACHEPREFIX only when O= is used.
Heh, I did a quick test here with Sphinx build. It Worked!
Patch is after my patch series, and it is just for Sphinx build.
We need something similar for the other places where the script is used.
Regards,
Mauro
---
[PATCH] docs: Makefile: store __pycache__ at the output directory
Instead of blocking creation of *.pyc cache, store python cache under Documentation/output/__pycache__
Documentation/output/__pycache__/
├── new_devel
│ └── v4l
│ └── docs
│ ├── Documentation
│ │ └── sphinx
│ │ ├── automarkup.cpython-313.pyc
│ │ ├── kernel_abi.cpython-313.pyc
│ │ ├── kerneldoc.cpython-313.pyc
│ │ ├── kernel_feat.cpython-313.pyc
│ │ ├── kernel_include.cpython-313.pyc
│ │ ├── kfigure.cpython-313.pyc
│ │ ├── load_config.cpython-313.pyc
│ │ ├── maintainers_include.cpython-313.pyc
│ │ ├── rstFlatTable.cpython-313.pyc
│ │ └── translations.cpython-313.pyc
│ └── scripts
│ └── lib
│ ├── abi
│ │ ├── abi_parser.cpython-313.pyc
│ │ └── helpers.cpython-313.pyc
│ └── kdoc
│ ├── kdoc_files.cpython-313.pyc
│ ├── kdoc_output.cpython-313.pyc
│ ├── kdoc_parser.cpython-313.pyc
│ └── kdoc_re.cpython-313.pyc
└── usr
├── lib
│ └── python3.13
│ └── site-packages
│ ├── abrt_exception_handler3.cpython-313.pyc
│ ├── alabaster
│ │ └── __init__.cpython-313.pyc
│ ├── babel
│ │ ├── core.cpython-313.pyc
│ │ ├── dates.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── localedata.cpython-313.pyc
│ │ ├── localtime
│ │ │ ├── _fallback.cpython-313.pyc
│ │ │ ├── _helpers.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ └── _unix.cpython-313.pyc
│ │ ├── messages
│ │ │ ├── catalog.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── mofile.cpython-313.pyc
│ │ │ ├── plurals.cpython-313.pyc
│ │ │ └── pofile.cpython-313.pyc
│ │ ├── plural.cpython-313.pyc
│ │ └── util.cpython-313.pyc
│ ├── chardet
│ │ ├── big5freq.cpython-313.pyc
│ │ ├── big5prober.cpython-313.pyc
│ │ ├── chardistribution.cpython-313.pyc
│ │ ├── charsetgroupprober.cpython-313.pyc
│ │ ├── charsetprober.cpython-313.pyc
│ │ ├── codingstatemachine.cpython-313.pyc
│ │ ├── codingstatemachinedict.cpython-313.pyc
│ │ ├── cp949prober.cpython-313.pyc
│ │ ├── enums.cpython-313.pyc
│ │ ├── escprober.cpython-313.pyc
│ │ ├── escsm.cpython-313.pyc
│ │ ├── eucjpprober.cpython-313.pyc
│ │ ├── euckrfreq.cpython-313.pyc
│ │ ├── euckrprober.cpython-313.pyc
│ │ ├── euctwfreq.cpython-313.pyc
│ │ ├── euctwprober.cpython-313.pyc
│ │ ├── gb2312freq.cpython-313.pyc
│ │ ├── gb2312prober.cpython-313.pyc
│ │ ├── hebrewprober.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── jisfreq.cpython-313.pyc
│ │ ├── johabfreq.cpython-313.pyc
│ │ ├── johabprober.cpython-313.pyc
│ │ ├── jpcntx.cpython-313.pyc
│ │ ├── langbulgarianmodel.cpython-313.pyc
│ │ ├── langgreekmodel.cpython-313.pyc
│ │ ├── langhebrewmodel.cpython-313.pyc
│ │ ├── langrussianmodel.cpython-313.pyc
│ │ ├── langthaimodel.cpython-313.pyc
│ │ ├── langturkishmodel.cpython-313.pyc
│ │ ├── latin1prober.cpython-313.pyc
│ │ ├── macromanprober.cpython-313.pyc
│ │ ├── mbcharsetprober.cpython-313.pyc
│ │ ├── mbcsgroupprober.cpython-313.pyc
│ │ ├── mbcssm.cpython-313.pyc
│ │ ├── resultdict.cpython-313.pyc
│ │ ├── sbcharsetprober.cpython-313.pyc
│ │ ├── sbcsgroupprober.cpython-313.pyc
│ │ ├── sjisprober.cpython-313.pyc
│ │ ├── universaldetector.cpython-313.pyc
│ │ ├── utf1632prober.cpython-313.pyc
│ │ ├── utf8prober.cpython-313.pyc
│ │ └── version.cpython-313.pyc
│ ├── charset_normalizer
│ │ ├── api.cpython-313.pyc
│ │ ├── cd.cpython-313.pyc
│ │ ├── constant.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── legacy.cpython-313.pyc
│ │ ├── md.cpython-313.pyc
│ │ ├── models.cpython-313.pyc
│ │ ├── utils.cpython-313.pyc
│ │ └── version.cpython-313.pyc
│ ├── defusedxml
│ │ ├── common.cpython-313.pyc
│ │ ├── ElementTree.cpython-313.pyc
│ │ └── __init__.cpython-313.pyc
│ ├── _distutils_hack
│ │ └── __init__.cpython-313.pyc
│ ├── docutils
│ │ ├── core.cpython-313.pyc
│ │ ├── frontend.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── io.cpython-313.pyc
│ │ ├── languages
│ │ │ ├── en.cpython-313.pyc
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── nodes.cpython-313.pyc
│ │ ├── parsers
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ └── rst
│ │ │ ├── directives
│ │ │ │ ├── admonitions.cpython-313.pyc
│ │ │ │ ├── body.cpython-313.pyc
│ │ │ │ ├── images.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ ├── misc.cpython-313.pyc
│ │ │ │ ├── parts.cpython-313.pyc
│ │ │ │ └── tables.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── languages
│ │ │ │ ├── en.cpython-313.pyc
│ │ │ │ └── __init__.cpython-313.pyc
│ │ │ ├── roles.cpython-313.pyc
│ │ │ ├── states.cpython-313.pyc
│ │ │ └── tableparser.cpython-313.pyc
│ │ ├── readers
│ │ │ ├── doctree.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ └── standalone.cpython-313.pyc
│ │ ├── statemachine.cpython-313.pyc
│ │ ├── transforms
│ │ │ ├── frontmatter.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── misc.cpython-313.pyc
│ │ │ ├── parts.cpython-313.pyc
│ │ │ ├── references.cpython-313.pyc
│ │ │ ├── universal.cpython-313.pyc
│ │ │ └── writer_aux.cpython-313.pyc
│ │ ├── utils
│ │ │ ├── code_analyzer.cpython-313.pyc
│ │ │ ├── error_reporting.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── math
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ ├── latex2mathml.cpython-313.pyc
│ │ │ │ ├── math2html.cpython-313.pyc
│ │ │ │ ├── tex2mathml_extern.cpython-313.pyc
│ │ │ │ ├── tex2unichar.cpython-313.pyc
│ │ │ │ └── unichar2tex.cpython-313.pyc
│ │ │ ├── punctuation_chars.cpython-313.pyc
│ │ │ ├── roman.cpython-313.pyc
│ │ │ ├── smartquotes.cpython-313.pyc
│ │ │ └── urischemes.cpython-313.pyc
│ │ └── writers
│ │ ├── docutils_xml.cpython-313.pyc
│ │ ├── html4css1
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── html5_polyglot
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── _html_base.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── latex2e
│ │ │ └── __init__.cpython-313.pyc
│ │ └── manpage.cpython-313.pyc
│ ├── idna
│ │ ├── core.cpython-313.pyc
│ │ ├── idnadata.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── intranges.cpython-313.pyc
│ │ └── package_data.cpython-313.pyc
│ ├── imagesize
│ │ ├── imagesize.cpython-313.pyc
│ │ └── __init__.cpython-313.pyc
│ ├── jinja2
│ │ ├── async_utils.cpython-313.pyc
│ │ ├── bccache.cpython-313.pyc
│ │ ├── compiler.cpython-313.pyc
│ │ ├── defaults.cpython-313.pyc
│ │ ├── environment.cpython-313.pyc
│ │ ├── exceptions.cpython-313.pyc
│ │ ├── ext.cpython-313.pyc
│ │ ├── filters.cpython-313.pyc
│ │ ├── _identifier.cpython-313.pyc
│ │ ├── idtracking.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── lexer.cpython-313.pyc
│ │ ├── loaders.cpython-313.pyc
│ │ ├── nodes.cpython-313.pyc
│ │ ├── optimizer.cpython-313.pyc
│ │ ├── parser.cpython-313.pyc
│ │ ├── runtime.cpython-313.pyc
│ │ ├── sandbox.cpython-313.pyc
│ │ ├── tests.cpython-313.pyc
│ │ ├── utils.cpython-313.pyc
│ │ └── visitor.cpython-313.pyc
│ ├── packaging
│ │ ├── _elffile.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── _manylinux.cpython-313.pyc
│ │ ├── _musllinux.cpython-313.pyc
│ │ ├── specifiers.cpython-313.pyc
│ │ ├── _structures.cpython-313.pyc
│ │ ├── tags.cpython-313.pyc
│ │ ├── utils.cpython-313.pyc
│ │ └── version.cpython-313.pyc
│ ├── pygments
│ │ ├── filter.cpython-313.pyc
│ │ ├── filters
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── formatter.cpython-313.pyc
│ │ ├── formatters
│ │ │ ├── html.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── latex.cpython-313.pyc
│ │ │ └── _mapping.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── lexer.cpython-313.pyc
│ │ ├── lexers
│ │ │ ├── c_cpp.cpython-313.pyc
│ │ │ ├── _css_builtins.cpython-313.pyc
│ │ │ ├── css.cpython-313.pyc
│ │ │ ├── data.cpython-313.pyc
│ │ │ ├── html.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── javascript.cpython-313.pyc
│ │ │ ├── jvm.cpython-313.pyc
│ │ │ ├── _lilypond_builtins.cpython-313.pyc
│ │ │ ├── lilypond.cpython-313.pyc
│ │ │ ├── lisp.cpython-313.pyc
│ │ │ ├── _mapping.cpython-313.pyc
│ │ │ ├── markup.cpython-313.pyc
│ │ │ ├── python.cpython-313.pyc
│ │ │ ├── ruby.cpython-313.pyc
│ │ │ ├── _scheme_builtins.cpython-313.pyc
│ │ │ └── special.cpython-313.pyc
│ │ ├── modeline.cpython-313.pyc
│ │ ├── plugin.cpython-313.pyc
│ │ ├── regexopt.cpython-313.pyc
│ │ ├── style.cpython-313.pyc
│ │ ├── styles
│ │ │ ├── friendly.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ └── _mapping.cpython-313.pyc
│ │ ├── token.cpython-313.pyc
│ │ ├── unistring.cpython-313.pyc
│ │ └── util.cpython-313.pyc
│ ├── pytz
│ │ ├── exceptions.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── lazy.cpython-313.pyc
│ │ ├── tzfile.cpython-313.pyc
│ │ └── tzinfo.cpython-313.pyc
│ ├── requests
│ │ ├── adapters.cpython-313.pyc
│ │ ├── api.cpython-313.pyc
│ │ ├── auth.cpython-313.pyc
│ │ ├── certs.cpython-313.pyc
│ │ ├── compat.cpython-313.pyc
│ │ ├── cookies.cpython-313.pyc
│ │ ├── exceptions.cpython-313.pyc
│ │ ├── hooks.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── _internal_utils.cpython-313.pyc
│ │ ├── models.cpython-313.pyc
│ │ ├── packages.cpython-313.pyc
│ │ ├── sessions.cpython-313.pyc
│ │ ├── status_codes.cpython-313.pyc
│ │ ├── structures.cpython-313.pyc
│ │ ├── utils.cpython-313.pyc
│ │ └── __version__.cpython-313.pyc
│ ├── snowballstemmer
│ │ ├── among.cpython-313.pyc
│ │ ├── arabic_stemmer.cpython-313.pyc
│ │ ├── armenian_stemmer.cpython-313.pyc
│ │ ├── basestemmer.cpython-313.pyc
│ │ ├── basque_stemmer.cpython-313.pyc
│ │ ├── catalan_stemmer.cpython-313.pyc
│ │ ├── danish_stemmer.cpython-313.pyc
│ │ ├── dutch_stemmer.cpython-313.pyc
│ │ ├── english_stemmer.cpython-313.pyc
│ │ ├── finnish_stemmer.cpython-313.pyc
│ │ ├── french_stemmer.cpython-313.pyc
│ │ ├── german_stemmer.cpython-313.pyc
│ │ ├── greek_stemmer.cpython-313.pyc
│ │ ├── hindi_stemmer.cpython-313.pyc
│ │ ├── hungarian_stemmer.cpython-313.pyc
│ │ ├── indonesian_stemmer.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── irish_stemmer.cpython-313.pyc
│ │ ├── italian_stemmer.cpython-313.pyc
│ │ ├── lithuanian_stemmer.cpython-313.pyc
│ │ ├── nepali_stemmer.cpython-313.pyc
│ │ ├── norwegian_stemmer.cpython-313.pyc
│ │ ├── porter_stemmer.cpython-313.pyc
│ │ ├── portuguese_stemmer.cpython-313.pyc
│ │ ├── romanian_stemmer.cpython-313.pyc
│ │ ├── russian_stemmer.cpython-313.pyc
│ │ ├── serbian_stemmer.cpython-313.pyc
│ │ ├── spanish_stemmer.cpython-313.pyc
│ │ ├── swedish_stemmer.cpython-313.pyc
│ │ ├── tamil_stemmer.cpython-313.pyc
│ │ ├── turkish_stemmer.cpython-313.pyc
│ │ └── yiddish_stemmer.cpython-313.pyc
│ ├── socks.cpython-313.pyc
│ ├── sphinx
│ │ ├── addnodes.cpython-313.pyc
│ │ ├── application.cpython-313.pyc
│ │ ├── builders
│ │ │ ├── changes.cpython-313.pyc
│ │ │ ├── dirhtml.cpython-313.pyc
│ │ │ ├── dummy.cpython-313.pyc
│ │ │ ├── epub3.cpython-313.pyc
│ │ │ ├── _epub_base.cpython-313.pyc
│ │ │ ├── gettext.cpython-313.pyc
│ │ │ ├── html
│ │ │ │ ├── _assets.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ └── transforms.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── latex
│ │ │ │ ├── constants.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ ├── nodes.cpython-313.pyc
│ │ │ │ ├── theming.cpython-313.pyc
│ │ │ │ ├── transforms.cpython-313.pyc
│ │ │ │ └── util.cpython-313.pyc
│ │ │ ├── linkcheck.cpython-313.pyc
│ │ │ ├── manpage.cpython-313.pyc
│ │ │ ├── singlehtml.cpython-313.pyc
│ │ │ ├── texinfo.cpython-313.pyc
│ │ │ ├── text.cpython-313.pyc
│ │ │ └── xml.cpython-313.pyc
│ │ ├── cmd
│ │ │ ├── build.cpython-313.pyc
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── config.cpython-313.pyc
│ │ ├── deprecation.cpython-313.pyc
│ │ ├── directives
│ │ │ ├── code.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── other.cpython-313.pyc
│ │ │ └── patches.cpython-313.pyc
│ │ ├── domains
│ │ │ ├── c
│ │ │ │ ├── _ast.cpython-313.pyc
│ │ │ │ ├── _ids.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ ├── _parser.cpython-313.pyc
│ │ │ │ └── _symbol.cpython-313.pyc
│ │ │ ├── changeset.cpython-313.pyc
│ │ │ ├── citation.cpython-313.pyc
│ │ │ ├── cpp
│ │ │ │ ├── _ast.cpython-313.pyc
│ │ │ │ ├── _ids.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ ├── _parser.cpython-313.pyc
│ │ │ │ └── _symbol.cpython-313.pyc
│ │ │ ├── index.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── javascript.cpython-313.pyc
│ │ │ ├── math.cpython-313.pyc
│ │ │ ├── python
│ │ │ │ ├── _annotations.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ └── _object.cpython-313.pyc
│ │ │ ├── rst.cpython-313.pyc
│ │ │ └── std
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── environment
│ │ │ ├── adapters
│ │ │ │ ├── asset.cpython-313.pyc
│ │ │ │ ├── indexentries.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ └── toctree.cpython-313.pyc
│ │ │ ├── collectors
│ │ │ │ ├── asset.cpython-313.pyc
│ │ │ │ ├── dependencies.cpython-313.pyc
│ │ │ │ ├── __init__.cpython-313.pyc
│ │ │ │ ├── metadata.cpython-313.pyc
│ │ │ │ ├── title.cpython-313.pyc
│ │ │ │ └── toctree.cpython-313.pyc
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── errors.cpython-313.pyc
│ │ ├── events.cpython-313.pyc
│ │ ├── ext
│ │ │ ├── autosectionlabel.cpython-313.pyc
│ │ │ ├── doctest.cpython-313.pyc
│ │ │ ├── ifconfig.cpython-313.pyc
│ │ │ ├── imgmath.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ └── mathjax.cpython-313.pyc
│ │ ├── extension.cpython-313.pyc
│ │ ├── highlighting.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── io.cpython-313.pyc
│ │ ├── jinja2glue.cpython-313.pyc
│ │ ├── locale
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── parsers.cpython-313.pyc
│ │ ├── project.cpython-313.pyc
│ │ ├── pycode
│ │ │ ├── ast.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ └── parser.cpython-313.pyc
│ │ ├── pygments_styles.cpython-313.pyc
│ │ ├── registry.cpython-313.pyc
│ │ ├── roles.cpython-313.pyc
│ │ ├── search
│ │ │ ├── en.cpython-313.pyc
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── theming.cpython-313.pyc
│ │ ├── transforms
│ │ │ ├── compact_bullet_list.cpython-313.pyc
│ │ │ ├── i18n.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── post_transforms
│ │ │ │ ├── code.cpython-313.pyc
│ │ │ │ ├── images.cpython-313.pyc
│ │ │ │ └── __init__.cpython-313.pyc
│ │ │ └── references.cpython-313.pyc
│ │ ├── util
│ │ │ ├── build_phase.cpython-313.pyc
│ │ │ ├── cfamily.cpython-313.pyc
│ │ │ ├── console.cpython-313.pyc
│ │ │ ├── display.cpython-313.pyc
│ │ │ ├── docfields.cpython-313.pyc
│ │ │ ├── docutils.cpython-313.pyc
│ │ │ ├── exceptions.cpython-313.pyc
│ │ │ ├── fileutil.cpython-313.pyc
│ │ │ ├── http_date.cpython-313.pyc
│ │ │ ├── i18n.cpython-313.pyc
│ │ │ ├── images.cpython-313.pyc
│ │ │ ├── index_entries.cpython-313.pyc
│ │ │ ├── __init__.cpython-313.pyc
│ │ │ ├── inspect.cpython-313.pyc
│ │ │ ├── inventory.cpython-313.pyc
│ │ │ ├── _io.cpython-313.pyc
│ │ │ ├── logging.cpython-313.pyc
│ │ │ ├── matching.cpython-313.pyc
│ │ │ ├── math.cpython-313.pyc
│ │ │ ├── nodes.cpython-313.pyc
│ │ │ ├── osutil.cpython-313.pyc
│ │ │ ├── parallel.cpython-313.pyc
│ │ │ ├── _pathlib.cpython-313.pyc
│ │ │ ├── png.cpython-313.pyc
│ │ │ ├── requests.cpython-313.pyc
│ │ │ ├── rst.cpython-313.pyc
│ │ │ ├── tags.cpython-313.pyc
│ │ │ ├── template.cpython-313.pyc
│ │ │ ├── texescape.cpython-313.pyc
│ │ │ └── typing.cpython-313.pyc
│ │ ├── versioning.cpython-313.pyc
│ │ └── writers
│ │ ├── html5.cpython-313.pyc
│ │ ├── html.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── latex.cpython-313.pyc
│ │ ├── manpage.cpython-313.pyc
│ │ ├── texinfo.cpython-313.pyc
│ │ ├── text.cpython-313.pyc
│ │ └── xml.cpython-313.pyc
│ ├── sphinxcontrib
│ │ ├── devhelp
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── htmlhelp
│ │ │ └── __init__.cpython-313.pyc
│ │ ├── qthelp
│ │ │ └── __init__.cpython-313.pyc
│ │ └── serializinghtml
│ │ ├── __init__.cpython-313.pyc
│ │ └── jsonimpl.cpython-313.pyc
│ └── urllib3
│ ├── _collections.cpython-313.pyc
│ ├── connection.cpython-313.pyc
│ ├── connectionpool.cpython-313.pyc
│ ├── contrib
│ │ ├── _appengine_environ.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ └── socks.cpython-313.pyc
│ ├── exceptions.cpython-313.pyc
│ ├── fields.cpython-313.pyc
│ ├── filepost.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── packages
│ │ ├── __init__.cpython-313.pyc
│ │ └── six.cpython-313.pyc
│ ├── poolmanager.cpython-313.pyc
│ ├── request.cpython-313.pyc
│ ├── response.cpython-313.pyc
│ ├── util
│ │ ├── connection.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── proxy.cpython-313.pyc
│ │ ├── queue.cpython-313.pyc
│ │ ├── request.cpython-313.pyc
│ │ ├── response.cpython-313.pyc
│ │ ├── retry.cpython-313.pyc
│ │ ├── ssl_.cpython-313.pyc
│ │ ├── ssl_match_hostname.cpython-313.pyc
│ │ ├── ssltransport.cpython-313.pyc
│ │ ├── timeout.cpython-313.pyc
│ │ ├── url.cpython-313.pyc
│ │ └── wait.cpython-313.pyc
│ └── _version.cpython-313.pyc
└── lib64
└── python3.13
├── argparse.cpython-313.pyc
├── ast.cpython-313.pyc
├── base64.cpython-313.pyc
├── bdb.cpython-313.pyc
├── bisect.cpython-313.pyc
├── bz2.cpython-313.pyc
├── calendar.cpython-313.pyc
├── cmd.cpython-313.pyc
├── code.cpython-313.pyc
├── codeop.cpython-313.pyc
├── collections
│ └── __init__.cpython-313.pyc
├── _colorize.cpython-313.pyc
├── _compat_pickle.cpython-313.pyc
├── _compression.cpython-313.pyc
├── configparser.cpython-313.pyc
├── contextlib.cpython-313.pyc
├── contextvars.cpython-313.pyc
├── copy.cpython-313.pyc
├── copyreg.cpython-313.pyc
├── csv.cpython-313.pyc
├── dataclasses.cpython-313.pyc
├── datetime.cpython-313.pyc
├── decimal.cpython-313.pyc
├── difflib.cpython-313.pyc
├── dis.cpython-313.pyc
├── doctest.cpython-313.pyc
├── email
│ ├── base64mime.cpython-313.pyc
│ ├── charset.cpython-313.pyc
│ ├── _encoded_words.cpython-313.pyc
│ ├── encoders.cpython-313.pyc
│ ├── errors.cpython-313.pyc
│ ├── feedparser.cpython-313.pyc
│ ├── header.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── iterators.cpython-313.pyc
│ ├── message.cpython-313.pyc
│ ├── _parseaddr.cpython-313.pyc
│ ├── parser.cpython-313.pyc
│ ├── _policybase.cpython-313.pyc
│ ├── quoprimime.cpython-313.pyc
│ └── utils.cpython-313.pyc
├── encodings
│ ├── aliases.cpython-313.pyc
│ ├── idna.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── unicode_escape.cpython-313.pyc
│ ├── utf_8.cpython-313.pyc
│ └── utf_8_sig.cpython-313.pyc
├── enum.cpython-313.pyc
├── filecmp.cpython-313.pyc
├── fnmatch.cpython-313.pyc
├── functools.cpython-313.pyc
├── __future__.cpython-313.pyc
├── gettext.cpython-313.pyc
├── glob.cpython-313.pyc
├── gzip.cpython-313.pyc
├── hashlib.cpython-313.pyc
├── heapq.cpython-313.pyc
├── hmac.cpython-313.pyc
├── html
│ ├── entities.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ └── parser.cpython-313.pyc
├── http
│ ├── client.cpython-313.pyc
│ ├── cookiejar.cpython-313.pyc
│ ├── cookies.cpython-313.pyc
│ └── __init__.cpython-313.pyc
├── importlib
│ ├── _abc.cpython-313.pyc
│ ├── abc.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── metadata
│ │ ├── _collections.cpython-313.pyc
│ │ ├── _functools.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── _itertools.cpython-313.pyc
│ │ └── _meta.cpython-313.pyc
│ └── resources
│ ├── abc.cpython-313.pyc
│ ├── _common.cpython-313.pyc
│ ├── _functional.cpython-313.pyc
│ └── __init__.cpython-313.pyc
├── inspect.cpython-313.pyc
├── ipaddress.cpython-313.pyc
├── json
│ ├── decoder.cpython-313.pyc
│ ├── encoder.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ └── scanner.cpython-313.pyc
├── keyword.cpython-313.pyc
├── linecache.cpython-313.pyc
├── locale.cpython-313.pyc
├── logging
│ ├── handlers.cpython-313.pyc
│ └── __init__.cpython-313.pyc
├── lzma.cpython-313.pyc
├── _markupbase.cpython-313.pyc
├── mimetypes.cpython-313.pyc
├── multiprocessing
│ ├── connection.cpython-313.pyc
│ ├── context.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── popen_fork.cpython-313.pyc
│ ├── process.cpython-313.pyc
│ ├── reduction.cpython-313.pyc
│ └── util.cpython-313.pyc
├── numbers.cpython-313.pyc
├── opcode.cpython-313.pyc
├── _opcode_metadata.cpython-313.pyc
├── operator.cpython-313.pyc
├── optparse.cpython-313.pyc
├── pathlib
│ ├── _abc.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ └── _local.cpython-313.pyc
├── pdb.cpython-313.pyc
├── pickle.cpython-313.pyc
├── platform.cpython-313.pyc
├── pprint.cpython-313.pyc
├── queue.cpython-313.pyc
├── quopri.cpython-313.pyc
├── random.cpython-313.pyc
├── re
│ ├── _casefix.cpython-313.pyc
│ ├── _compiler.cpython-313.pyc
│ ├── _constants.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ └── _parser.cpython-313.pyc
├── reprlib.cpython-313.pyc
├── rlcompleter.cpython-313.pyc
├── selectors.cpython-313.pyc
├── shutil.cpython-313.pyc
├── signal.cpython-313.pyc
├── site-packages
│ ├── markupsafe
│ │ └── __init__.cpython-313.pyc
│ └── PIL
│ ├── _binary.cpython-313.pyc
│ ├── _deprecate.cpython-313.pyc
│ ├── ExifTags.cpython-313.pyc
│ ├── Image.cpython-313.pyc
│ ├── ImageMode.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── TiffTags.cpython-313.pyc
│ ├── _typing.cpython-313.pyc
│ ├── _util.cpython-313.pyc
│ └── _version.cpython-313.pyc
├── socket.cpython-313.pyc
├── ssl.cpython-313.pyc
├── string.cpython-313.pyc
├── stringprep.cpython-313.pyc
├── struct.cpython-313.pyc
├── subprocess.cpython-313.pyc
├── sysconfig
│ └── __init__.cpython-313.pyc
├── _sysconfigdata__linux_x86_64-linux-gnu.cpython-313.pyc
├── tempfile.cpython-313.pyc
├── textwrap.cpython-313.pyc
├── threading.cpython-313.pyc
├── token.cpython-313.pyc
├── tokenize.cpython-313.pyc
├── tomllib
│ ├── __init__.cpython-313.pyc
│ ├── _parser.cpython-313.pyc
│ ├── _re.cpython-313.pyc
│ └── _types.cpython-313.pyc
├── traceback.cpython-313.pyc
├── types.cpython-313.pyc
├── typing.cpython-313.pyc
├── unittest
│ ├── case.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── loader.cpython-313.pyc
│ ├── main.cpython-313.pyc
│ ├── result.cpython-313.pyc
│ ├── runner.cpython-313.pyc
│ ├── signals.cpython-313.pyc
│ ├── suite.cpython-313.pyc
│ └── util.cpython-313.pyc
├── urllib
│ ├── error.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── parse.cpython-313.pyc
│ ├── request.cpython-313.pyc
│ └── response.cpython-313.pyc
├── uuid.cpython-313.pyc
├── warnings.cpython-313.pyc
├── weakref.cpython-313.pyc
├── _weakrefset.cpython-313.pyc
├── xml
│ ├── dom
│ │ ├── domreg.cpython-313.pyc
│ │ ├── __init__.cpython-313.pyc
│ │ ├── minicompat.cpython-313.pyc
│ │ ├── minidom.cpython-313.pyc
│ │ ├── NodeFilter.cpython-313.pyc
│ │ └── xmlbuilder.cpython-313.pyc
│ ├── etree
│ │ ├── ElementPath.cpython-313.pyc
│ │ ├── ElementTree.cpython-313.pyc
│ │ └── __init__.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── parsers
│ │ ├── expat.cpython-313.pyc
│ │ └── __init__.cpython-313.pyc
│ └── sax
│ ├── _exceptions.cpython-313.pyc
│ ├── expatreader.cpython-313.pyc
│ ├── handler.cpython-313.pyc
│ ├── __init__.cpython-313.pyc
│ ├── saxutils.cpython-313.pyc
│ └── xmlreader.cpython-313.pyc
├── zipfile
│ ├── __init__.cpython-313.pyc
│ └── _path
│ ├── glob.cpython-313.pyc
│ └── __init__.cpython-313.pyc
└── zoneinfo
├── _common.cpython-313.pyc
├── __init__.cpython-313.pyc
└── _tzpath.cpython-313.pyc109 directories, 634 files
Documentation/Makefile +1 -1
@@ -84,7 +84,7 @@ loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit; quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media $2 && \ - PYTHONDONTWRITEBYTECODE=1 \ + PYTHONPYCACHEPREFIX=$(abspath $(BUILDDIR)/__pycache__) \ BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \ $(PYTHON3) $(srctree)/scripts/jobserver-exec \ $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
Quoted 5 lines · Akira Yokosawa
Quoted 4 lines · Andy Shevchenko
Quoted 3 lines · Akira Yokosawa
Quoted 2 lines · Andy Shevchenko
Quoted 1 line · Jonathan Corbet
Dmitry Baryshkov <[email protected]> writes:
[...]
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
Quoted 8 lines
Quoted 2 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
It's not so easy. The Python is written in a way that it thinks it will never runs object files separately from the source. Hence that variable sets only the folder per script as _home_ for the cache. It's completely unusable. They took it wrong. It still can be _painfully_ used, but it will make Makefiles uglier.
But, PYTHONPYCACHEPREFIX can be set as an environment variable.
For example, try:
export PYTHONPYCACHEPREFIX="$HOME/.cache/__pycache__"
Wouldn't it be good enough for you?
Quoted 8 lines
Quoted 4 lines
Of course not. We have _many_ scripts in python in kernel and having a cache there for _all_ of them is simply WRONG. You never know what clashes can be there with two complicated enough scripts which may have same module names, etc.
Interesting...
I'm suspecting you replied without having tried the setting...
I tried before, but I admit, that I have missed something. It was a mess in that case. Now I probably can't repeat as I don't remember what was the environment and settings I had that time. I'm really glad to see it is working this way!
Signature · 2 lines
With Best Regards, Andy Shevchenko
Em Mon, 21 Apr 2025 10:35:29 -0600
Jonathan Corbet <[email protected]> escreveu:
Quoted 11 lines
Quoted 9 lines · Dmitry Baryshkov
Quoted 7 lines · Mauro Carvalho Chehab
As reported by Andy, the Kernel build system runs kernel-doc script for DRM, when W=1. Due to Python's normal behavior, its JIT compiler will create a bytecode and store it under
scripts/lib/*/__pycache__.As one may be using O= and even having the sources on a read-only mount point, disable its creation during build time.
Quoted 7 lines
Quoted 2 lines
Would it be possible to properly support O= and create pyc / pycache inside the object/output dir?
I have to confess, I've been wondering if we should be treating the .pyc files like we treat .o files or other intermediate products. Rather than trying to avoid their creation entirely, perhaps we should just be sure they end up in the right place and are properly cleaned up...?
I sent a v4 of the series disabling *.pyc creation, basically placing *.pyc at the right place on .gitignore.
That's said, I agree that the best would be to use PYTHONPYCACHEPREFIX, placing the intermediate products altogether with O= results. There is however something we need to deal with. To follow our building system to the letter, if we have *.py files at scripts/lib/kdoc, the intermediate files should be at: either:
- {outdir}/scripts/lib/kdoc; or:
- {outdir}/scripts/lib/kdoc/__pycache__
The same shall happen for all other places including Documentation/sphinx. In the specific case of Sphinx extensions, it would mean that it would produce *.pyc for both scripts/lib/*/*.py and Documentation/sphinx. No idea how to make Python to do that without doing some horrible tricks.
An alternate approach would be to store all of them at the same place, like:
- {outdir}/__pycache__
The problem is that, if we end having two scripts with the same name, and using PYTHONPYCACHEPREFIX would place both of them at the same place, we'll have troubles.
IMO, let's apply this series first, and then see if we can replace patch 3 with another one using PYTHONPYCACHEPREFIX, after we're sure that there's a way for it to do the right thing.
Quoted 3 lines
To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX should do the trick?
Regards,
Mauro