Missing .pyc files in target filesystem

We're not getting python .pyc files for the Python packages in the target filesystem /usr/lib/python3.8/ and I can't figure out why, despite several days of looking at this.

We're on Buildroot 2020.02.03 and I've tried both

BR2_PACKAGE_PYTHON_PYC_ONLY=y

BR2_PACKAGE_PYTHON_PY_PYC=y

in the config. I only see .py files.

Any ideas on how to debug this? What's the thing in buildroot that creates the .pyc files?

Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20210412/f739d2a2/attachment.html>

How did you generate this config? The symbols are

BR2_PACKAGE_PYTHON3_PYC_ONLY
BR2_PACKAGE_PYTHON3_PY_PYC

when you're using python3. Setting the python2 configuration will not affect python3 - and in fact, those python2 symbols should be removed when any kconfig target is run.

Regards,
Arnout

Thanks, Arnout. We were in fact using the symbols

BR2_PACKAGE_PYTHON3_PYC_ONLY
BR2_PACKAGE_PYTHON3_PY_PYC

When I typed it into my email, I left off the 3. I checked my config. The last build I tried had

BR2_PACKAGE_PYTHON3_PYC_ONLY

And we got .py files in the target filesystem

-----Original Message-----
From: Arnout Vandecappelle <[email protected]>
Sent: Monday, April 12, 2021 12:50 PM
To: swirsky at gmail.com; buildroot at busybox.net
Subject: Re: [Buildroot] Missing .pyc files in target filesystem

How did you generate this config? The symbols are

BR2_PACKAGE_PYTHON3_PYC_ONLY
BR2_PACKAGE_PYTHON3_PY_PYC

when you're using python3. Setting the python2 configuration will not affect python3 - and in fact, those python2 symbols should be removed when any kconfig target is run.

Regards,
Arnout

Ah, too bad :-)

The code that generates the pyc files is the following:

define PYTHON3_CREATE_PYC_FILES

        $(PYTHON3_FIX_TIME)
        PYTHONPATH="$(PYTHON3_PATH)" \
        $(HOST_DIR)/bin/python$(PYTHON3_VERSION_MAJOR) \
                $(TOPDIR)/support/scripts/pycompile.py \
                $(if $(VERBOSE),--verbose) \
                --strip-root $(TARGET_DIR) \
                $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)

endef

So apparently pycompile doesn't find the .py files?

You can try building with "make V=1" and see if pycompile reports anything.

I've just tried the BR2_PACKAGE_PYTHON3_PYC_ONLY locally and I got only .pyc
(and .whl) files.

It's really surprising you have .py files at all, because the following is also
there:

ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
define PYTHON3_REMOVE_PY_FILES

        find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' \
                $(if $(strip $(KEEP_PYTHON_PY_FILES)),-not \( $(call
finddirclauses,$(TARGET_DIR),$(KEEP_PYTHON_PY_FILES)) \) ) \
                -print0 | \
                xargs -0 --no-run-if-empty rm -f

endef
PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES endif

Regards,
Arnout

I never did figure out the why, but I "binary searched" the past dozen or so commits and found the one that broke the .pyc files. It was when I added a certain external out-of-tree package to our project.

What's interesting is
- this package was not specified in the standard config file (ie., the files under configs that the menu manages). It was listed in the Config.in where all the external packages are enumerated. I was under the impression that packages not in a ~/project/configs configuration had no effect on the build
- the package in question was "remote_pdb"

I thought that, maybe, someone put some code to see if you've turned on any form of pdb and suppresses .pyc generation but I saw no such thing when reading the source code. So my best guess is that I had a faulty makefile for this external project that was causing the rest of the python machinery to fail (perhaps none of the other "hooks" ran) and it failed without letting me know.

Thanks for all your help, Arnout!