Don't create Python bytecode when building the kernel

PATCH Superseded Mauro Carvalho Chehab 3 participants 8 messages Apr 16, 2025

Subsystems (4)

DRM drivers Documentation DRM misc Intel i915

lore

Share & tools

lore

Mauro Carvalho Chehab

author

2025-04-16 06:58 UTC #0 lore raw

v2 cover Don't create Python bytecode when building the kernel

As reported by Andy, the Kernel build system runs kernel-doc script for DRM, when W=1. Due to Python's normal behavior, its JIT compiler will create a bytecode and store it under scripts/lib/*/__pycache__.

As one may be using O= and even having the sources on a read-only mount point, disable its creation during build time.

This only solves half of the issue though, as one may be manually running the script by hand, without asking Python to not store any bytecode. This should be OK, but afterwards, git status will list the __pycache__ as not committed. To prevent that, add *.pyc to .gitignore.

Mauro Carvalho Chehab (2):
scripts/kernel-doc.py: don't create *.pyc files
.gitignore: ignore Python compiled bytecode

 .gitignore                    | 1 +
 drivers/gpu/drm/Makefile      | 2 +-
 drivers/gpu/drm/i915/Makefile | 2 +-
 include/drm/Makefile          | 2 +-
 scripts/find-unused-docs.sh   | 2 +-
 5 files changed, 5 insertions(+), 4 deletions(-)

Signature · 1 line

2.49.0

Mauro Carvalho Chehab

author

2025-04-16 06:58 UTC #1 lore raw

PATCH v2 1/2 scripts/kernel-doc.py: don't create *.pyc files

As reported by Andy, kernel-doc.py is creating a __pycache__ directory at build time.

Disable creation of __pycache__ for the libraries used by kernel-doc.py, when excecuted via the build system or via scripts/find-unused-docs.sh.

Reported-by Andy Shevchenko <[email protected]>

Closes https://lore.kernel.org/linux-doc/[email protected]/ open here

Signed-off-by Mauro Carvalho Chehab <[email protected]>

---
 drivers/gpu/drm/Makefile      | 2 +-
 drivers/gpu/drm/i915/Makefile | 2 +-
 include/drm/Makefile          | 2 +-
 scripts/find-unused-docs.sh   | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

drivers/gpu/drm/Makefile +1 -1

index ed54a546bbe2..1469d64f8783 100644
@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \
 quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@)
       cmd_hdrtest = \
 		$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \
-		$(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
+		$(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
 		touch $@
 
 $(obj)/%.hdrtest: $(src)/%.h FORCE

drivers/gpu/drm/i915/Makefile +1 -1

index ed05b131ed3a..bb873f9cc2aa 100644
@@ -408,7 +408,7 @@ obj-$(CONFIG_DRM_I915_GVT_KVMGT) += kvmgt.o
 #
 # Enable locally for CONFIG_DRM_I915_WERROR=y. See also scripts/Makefile.build
 ifdef CONFIG_DRM_I915_WERROR
-    cmd_checkdoc = $(srctree)/scripts/kernel-doc -none -Werror $<
+    cmd_checkdoc = $(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none -Werror $<
 endif
 
 # header test

include/drm/Makefile +1 -1

index a7bd15d2803e..6088ea458f44 100644
@@ -11,7 +11,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \
 quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@)
       cmd_hdrtest = \
 		$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \
-		$(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
+		$(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
 		touch $@
 
 $(obj)/%.hdrtest: $(src)/%.h FORCE

scripts/find-unused-docs.sh +1 -1

index ee6a50e33aba..d6d397fbf917 100755
@@ -54,7 +54,7 @@ for file in `find $1 -name '*.c'`; do
 	if [[ ${FILES_INCLUDED[$file]+_} ]]; then
 	continue;
 	fi
-	str=$(scripts/kernel-doc -export "$file" 2>/dev/null)
+	str=$(PYTHONDONTWRITEBYTECODE=1 scripts/kernel-doc -export "$file" 2>/dev/null)
 	if [[ -n "$str" ]]; then
 	echo "$file"
 	fi

Signature · 1 line

2.49.0

Andy Shevchenko

2025-04-16 07:22 UTC #2 lore raw

Re: [PATCH v2 1/2] scripts/kernel-doc.py: don't create *.pyc files

Quoted 6 lines · Mauro Carvalho Chehab

As reported by Andy, kernel-doc.py is creating a __pycache__ directory at build time.
Disable creation of __pycache__ for the libraries used by kernel-doc.py, when excecuted via the build system or via scripts/find-unused-docs.sh.

Nope, still have it.

I used today's Linux Next with some local code patches (unrelated to any scripts or kernel doc or python).

To confirm, I even done again a clean build.

Signature · 2 lines

With Best Regards,
Andy Shevchenko

Mauro Carvalho Chehab

author

2025-04-16 07:52 UTC #3 lore raw

Re: [PATCH v2 1/2] scripts/kernel-doc.py: don't create *.pyc files

Em Wed, 16 Apr 2025 10:22:24 +0300
Andy Shevchenko <[email protected]> escreveu:

Quoted 8 lines · Mauro Carvalho Chehab

Quoted 6 lines
As reported by Andy, kernel-doc.py is creating a __pycache__ directory at build time.
Disable creation of __pycache__ for the libraries used by kernel-doc.py, when excecuted via the build system or via scripts/find-unused-docs.sh.

Quoted 7 lines

Nope, still have it.
I used today's Linux Next with some local code patches (unrelated to any scripts or kernel doc or python).
To confirm, I even done again a clean build.

There were one missing place.

Sent a v3. Please test.

Regards,
Mauro

Jani Nikula

maintainer

2025-04-16 07:33 UTC #4 lore raw

Re: [PATCH v2 1/2] scripts/kernel-doc.py: don't create *.pyc files

In reply to Mauro Carvalho Chehab

Quoted 22 lines · Mauro Carvalho Chehab

As reported by Andy, kernel-doc.py is creating a __pycache__ directory at build time.
Disable creation of __pycache__ for the libraries used by kernel-doc.py, when excecuted via the build system or via scripts/find-unused-docs.sh.
Reported-by Andy Shevchenko <[email protected]>
Closes https://lore.kernel.org/linux-doc/[email protected]/ open here
Signed-off-by Mauro Carvalho Chehab <[email protected]>
---
 drivers/gpu/drm/Makefile      | 2 +-
 drivers/gpu/drm/i915/Makefile | 2 +-
 include/drm/Makefile          | 2 +-
 scripts/find-unused-docs.sh   | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
drivers/gpu/drm/Makefile +0 -0
index ed54a546bbe2..1469d64f8783 100644

drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239

Quoted 5 lines

drivers/gpu/drm/Makefile +1 -1

@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \
 quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@)
       cmd_hdrtest = \
 		$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \
-		$(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
+		$(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \

It doesn't work to put PYTHONDONTWRITEBYTECODE=1 as a parameter to kernel-doc...

BR,
Jani.

Quoted 42 lines

drivers/gpu/drm/Makefile +0 -0

 		touch $@
 
 $(obj)/%.hdrtest: $(src)/%.h FORCE

drivers/gpu/drm/i915/Makefile +1 -1

index ed05b131ed3a..bb873f9cc2aa 100644
@@ -408,7 +408,7 @@ obj-$(CONFIG_DRM_I915_GVT_KVMGT) += kvmgt.o
 #
 # Enable locally for CONFIG_DRM_I915_WERROR=y. See also scripts/Makefile.build
 ifdef CONFIG_DRM_I915_WERROR
-    cmd_checkdoc = $(srctree)/scripts/kernel-doc -none -Werror $<
+    cmd_checkdoc = $(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none -Werror $<
 endif
 
 # header test

include/drm/Makefile +1 -1

index a7bd15d2803e..6088ea458f44 100644
@@ -11,7 +11,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \
 quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@)
       cmd_hdrtest = \
 		$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \
-		$(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
+		$(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
 		touch $@
 
 $(obj)/%.hdrtest: $(src)/%.h FORCE

scripts/find-unused-docs.sh +1 -1

index ee6a50e33aba..d6d397fbf917 100755
@@ -54,7 +54,7 @@ for file in `find $1 -name '*.c'`; do
 	if [[ ${FILES_INCLUDED[$file]+_} ]]; then
 	continue;
 	fi
-	str=$(scripts/kernel-doc -export "$file" 2>/dev/null)
+	str=$(PYTHONDONTWRITEBYTECODE=1 scripts/kernel-doc -export "$file" 2>/dev/null)
 	if [[ -n "$str" ]]; then
 	echo "$file"
 	fi

Signature · 1 line

Jani Nikula, Intel

Jani Nikula

maintainer

2025-04-16 07:36 UTC #5 lore raw

Re: [PATCH v2 1/2] scripts/kernel-doc.py: don't create *.pyc files

Quoted 18 lines · Jani Nikula

Quoted 17 lines · Mauro Carvalho Chehab
As reported by Andy, kernel-doc.py is creating a __pycache__ directory at build time.
Disable creation of __pycache__ for the libraries used by kernel-doc.py, when excecuted via the build system or via scripts/find-unused-docs.sh.
Reported-by Andy Shevchenko <[email protected]>
Closes https://lore.kernel.org/linux-doc/[email protected]/ open here
Signed-off-by Mauro Carvalho Chehab <[email protected]>
---
 drivers/gpu/drm/Makefile      | 2 +-
 drivers/gpu/drm/i915/Makefile | 2 +-
 include/drm/Makefile          | 2 +-
 scripts/find-unused-docs.sh   | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

Quoted 13 lines

drivers/gpu/drm/Makefile always-$(CONFIG_DRM_HEADER_TEST) += \ · line 239

Quoted 10 lines

drivers/gpu/drm/Makefile +1 -1

index ed54a546bbe2..1469d64f8783 100644
@@ -236,7 +236,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \
 quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@)
       cmd_hdrtest = \
 		$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \
-		$(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
+		$(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \

It doesn't work to put PYTHONDONTWRITEBYTECODE=1 as a parameter to kernel-doc...

Moreover, KERNELDOC is only defined in Documentation/Makefile. It's empty here.

Also scripts/Makefile.build uses kernel-doc, which is probably the one creating __pycache__ Andy sees.

Quoted 47 lines

BR,
Jani.

Quoted 42 lines

drivers/gpu/drm/Makefile +0 -0

 		touch $@
 
 $(obj)/%.hdrtest: $(src)/%.h FORCE

drivers/gpu/drm/i915/Makefile +1 -1

index ed05b131ed3a..bb873f9cc2aa 100644
@@ -408,7 +408,7 @@ obj-$(CONFIG_DRM_I915_GVT_KVMGT) += kvmgt.o
 #
 # Enable locally for CONFIG_DRM_I915_WERROR=y. See also scripts/Makefile.build
 ifdef CONFIG_DRM_I915_WERROR
-    cmd_checkdoc = $(srctree)/scripts/kernel-doc -none -Werror $<
+    cmd_checkdoc = $(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none -Werror $<
 endif
 
 # header test

include/drm/Makefile +1 -1

index a7bd15d2803e..6088ea458f44 100644
@@ -11,7 +11,7 @@ always-$(CONFIG_DRM_HEADER_TEST) += \
 quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@)
       cmd_hdrtest = \
 		$(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \
-		$(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
+		$(KERNELDOC) PYTHONDONTWRITEBYTECODE=1 -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \
 		touch $@
 
 $(obj)/%.hdrtest: $(src)/%.h FORCE

scripts/find-unused-docs.sh +1 -1

index ee6a50e33aba..d6d397fbf917 100755
@@ -54,7 +54,7 @@ for file in `find $1 -name '*.c'`; do
 	if [[ ${FILES_INCLUDED[$file]+_} ]]; then
 	continue;
 	fi
-	str=$(scripts/kernel-doc -export "$file" 2>/dev/null)
+	str=$(PYTHONDONTWRITEBYTECODE=1 scripts/kernel-doc -export "$file" 2>/dev/null)
 	if [[ -n "$str" ]]; then
 	echo "$file"
 	fi

Signature · 1 line

Jani Nikula, Intel

Mauro Carvalho Chehab

author

2025-04-16 06:58 UTC #6 lore raw

PATCH v2 2/2 .gitignore: ignore Python compiled bytecode

In reply to Mauro Carvalho Chehab

While the building system doesn't create any Python JIT bytecode, if one manually runs kernel-doc.py or get_abi.py, Python will, by default, create a bytecode and store it under scripts/lib/*.

This is normal, and not controlled by the Kernel itself. So, add *.pyc as an extension to be ignored.

Signed-off-by Mauro Carvalho Chehab <[email protected]>

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

.gitignore +1 -0

index f2f63e47fb88..2e01bc0b2d2a 100644
@@ -39,6 +39,7 @@
 *.mod.c
 *.o
 *.o.*
+*.pyc
 *.patch
 *.rmeta
 *.rpm

Signature · 1 line

2.49.0

Andy Shevchenko

2025-04-16 07:16 UTC #7 lore raw

Re: [PATCH v2 2/2] .gitignore: ignore Python compiled bytecode

Quoted 6 lines · Mauro Carvalho Chehab

While the building system doesn't create any Python JIT bytecode, if one manually runs kernel-doc.py or get_abi.py, Python will, by default, create a bytecode and store it under scripts/lib/*.
This is normal, and not controlled by the Kernel itself. So, add *.pyc as an extension to be ignored.

...

Quoted 4 lines

*.mod.c
*.o
*.o.*
+*.pyc

Shouldn't we prevent the order? (This should go after the "patch")

Quoted 3 lines

*.patch
*.rmeta
*.rpm

...

P.S. I'm going to test the first patch (at least) soon.

Signature · 2 lines

With Best Regards,
Andy Shevchenko