support/size-stats: actually handle .pyc files
cover support/size-stats: actually handle .pyc files
From: "Yann E. MORIN" <[email protected]>
Hello All!
This small two-patch series properly associates .pyc files to the corresponding package.
Regards,
Yann E. MORIN.
The following changes since commit 4cc29326c91877a385eeb288378c1099ea240ee9
Merge branch 'nyma/misc-fixes' into orange-master (2026-07-01 08:13:38 +0200)
are available as patches in this mail series,
for you to apply patches up to c45315df9f1084fe0f6f932e1523cfa9c4e6d992
support/scripts/size-stats: properly assign pre-compiled .pyc files (2026-08-10 15:24:11 +0200)
----------------------------------------------------------------
Yann E. MORIN (2):
support/scripts/size-stats: prepare for more .py associated files support/scripts/size-stats: properly assign pre-compiled .pyc files
support/scripts/size-stats | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-)
Signature · 24 lines
____________ .-----------------.--------------------: _ :------------------. | Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON | | | Software Designer | _/ - /' | \ / CAMPAIGN | | +33 638.411.245 '--------------------: (_ `--, | X AGAINST | | yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL | '--------------------------------------:______/_____:------------------' ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 1/2 support/scripts/size-stats: prepare for more .py associated files
From: "Yann E. MORIN" <[email protected]>
--- support/scripts/size-stats | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
support/scripts/size-stats +18 -3
@@ -54,9 +54,8 @@ class Config: # pkg: package to which the file belongs # def add_file(filesdict, relpath, abspath, pkg): - if relpath.endswith(".py"): - # also check for compiled .pyc file - add_file(filesdict, relpath + "c", abspath + "c", pkg) + for relpath2, abspath2 in get_associated_files(relpath, abspath): + add_file(filesdict, relpath2, abspath2, pkg) if not os.path.exists(abspath): return if os.path.islink(abspath):
@@ -65,6 +64,22 @@ def add_file(filesdict, relpath, abspath, pkg): filesdict[relpath] = (pkg, sz) +# +# Returns an iterator over the files associated with the given one +# as a 2-tuple of (relpath, abspath). +# +# Supported files: +# *.py => yields the list of corresponding pre-compiled .pyc files +# +# relpath: relative path of the file +# fullpath: absolute path to the file +# +def get_associated_files(relpath, abspath): + # also check for compiled .pyc file + if relpath.endswith(".py"): + yield (relpath + "c", abspath + "c") + + # # This function returns a dict where each key is the path of a file in # the root filesystem, and the value is a tuple containing two
Signature · 18 lines
2.43.0 ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/2 support/scripts/size-stats: properly assign pre-compiled .pyc files
In reply to [email protected]From: "Yann E. MORIN" <[email protected]>
Since commit 3fed42456693 (package/python3: use the provided pyc compiler), we abide by the PEP 3147 guidelines on where the pyc files are located. In practice, this means that, for a .py file /foo/bar.py, the corresponding .pyc file will be /foo/__pycache__/bar.pythonX.Y.pyc (where X.Y is the current major.minor python version, e.g. 3.14).
Add support for this layout in the size-stats scrip, which so far only supported the legacy, pre-PEP 3147 layout of having the .pyc next to the .py.
When we associate files to a package, we only have the original name, and we must construct the associated filenames. For the .pyc, this would require that the version of python be passed to the size-stats script, but this is not so nice. Instead, just scan the pycache directory to find the corresponding files. This is not much nicer either, but at least the wart, if bigger, is in a single place that would be easy to chop out or rework if need be (e.g. if we need to add support for more similarly target-finalize-generated files in the future)...
--- support/scripts/size-stats | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
support/scripts/size-stats +16 -0
@@ -77,7 +77,23 @@ def add_file(filesdict, relpath, abspath, pkg): def get_associated_files(relpath, abspath): # also check for compiled .pyc file if relpath.endswith(".py"): + # Legacy .pyc next to .py yield (relpath + "c", abspath + "c") + # PEP 3147 layout + basename = os.path.basename(abspath).rsplit(".", 1)[0] + reldir = os.path.join(os.path.dirname(relpath), "__pycache__") + absdir = os.path.join(os.path.dirname(abspath), "__pycache__") + try: + for fname in ( + fn + for fn in os.listdir(absdir) + if fn.endswith(".pyc") + and fn.rsplit(".", 2)[0] == basename + ): + yield (os.path.join(reldir, fname), os.path.join(absdir, fname)) + except FileNotFoundError: + # No precompiled .pyc files + pass #
Signature · 18 lines
2.43.0 ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc files
Yann,
Quoted 47 lines
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, August 10, 2026 8:28 AM
To: [email protected]
Cc: [email protected]; Michael Klein <[email protected]>; Vincent Fazio <[email protected]>
Subject: [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc filesFrom: "Yann E. MORIN" <[email protected]>
Since commit
3fed42456693(package/python3: use the provided pyc compiler), we abide by the PEP 3147 guidelines on where the pyc files are located. In practice, this means that, for a .py file /foo/bar.py, the corresponding .pyc file will be /foo/__pycache__/bar.pythonX.Y.pyc (where X.Y is the current major.minor python version, e.g. 3.14).Add support for this layout in the size-stats scrip, which so far only supported the legacy, pre-PEP 3147 layout of having the .pyc next to the .py.
When we associate files to a package, we only have the original name, and we must construct the associated filenames. For the .pyc, this would require that the version of python be passed to the size-stats script, but this is not so nice. Instead, just scan the pycache directory to find the corresponding files. This is not much nicer either, but at least the wart, if bigger, is in a single place that would be easy to chop out or rework if need be (e.g. if we need to add support for more similarly target-finalize-generated files in the future)...
Signed-off-by Yann E. MORIN <[email protected]>Cc Michael Klein <[email protected]>Cc Vincent Fazio <[email protected]>--- support/scripts/size-stats | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)support/scripts/size-stats +1
-0@@ -77,7 +77,23 @@ def add_file(filesdict, relpath, abspath, pkg): def get_associated_files(relpath, abspath): # also check for compiled .pyc file if relpath.endswith(".py"): + # Legacy .pyc next to .py yield (relpath + "c", abspath + "c")
support/scripts/size-stats def add_file(filesdict, relpath, abspath, pkg): · line 91
Quoted 10 lines
support/scripts/size-stats +10
-0+ # PEP 3147 layout + basename = os.path.basename(abspath).rsplit(".", 1)[0] + reldir = os.path.join(os.path.dirname(relpath), "__pycache__") + absdir = os.path.join(os.path.dirname(abspath), "__pycache__") + try: + for fname in ( + fn + for fn in os.listdir(absdir) + if fn.endswith(".pyc") + and fn.rsplit(".", 2)[0] == basename
It's not an immediate concern because we don't have support for it (yet), but there are multiple optimization levels which pyc can be compiled for (1-3) [0][1].
I think (untested) the current rsplit will fail on an optimized file pattern and we can be a bit more future-proof so we don't accidentally run into this same problem again?
Maybe we can leverage source_from_cache [2] and compare the file names?
It's probably overkill, but I think this is one of those easily overlooked things where if we did add support for optimization levels that we'd miss fixing the code here to account for it.
Anyway, something to consider.
-Vincent
[0]: https://peps.python.org/pep-0488/
[1]: https://docs.python.org/3.15/using/cmdline.html#cmdoption-O
[2]: https://github.com/python/cpython/blob/20e6c2fc7c174342214d561845419c4030f8638f/Lib/importlib/_bootstrap_external.py#L299
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc files
Vincent, All,
Thanks for the review! :-)
On 2026-08-19 13:44 +0000, Vincent Fazio spake thusly:
Quoted 9 lines
Quoted 9 lines
Since commit
3fed42456693(package/python3: use the provided pyc compiler), we abide by the PEP 3147 guidelines on where the pyc files are located. In practice, this means that, for a .py file /foo/bar.py, the corresponding .pyc file will be /foo/__pycache__/bar.pythonX.Y.pyc (where X.Y is the current major.minor python version, e.g. 3.14).Add support for this layout in the size-stats scrip, which so far only supported the legacy, pre-PEP 3147 layout of having the .pyc next to the .py.
[--SNIP--]
Quoted 13 lines
support/scripts/size-stats def add_file(filesdict, relpath, abspath, pkg): · line 91
Quoted 12 lines
support/scripts/size-stats +11
-0+ # Legacy .pyc next to .py yield (relpath + "c", abspath + "c") + # PEP 3147 layout + basename = os.path.basename(abspath).rsplit(".", 1)[0] + reldir = os.path.join(os.path.dirname(relpath), "__pycache__") + absdir = os.path.join(os.path.dirname(abspath), "__pycache__") + try: + for fname in ( + fn + for fn in os.listdir(absdir) + if fn.endswith(".pyc") + and fn.rsplit(".", 2)[0] == basename
Quoted 6 lines
It's not an immediate concern because we don't have support for it (yet), but there are multiple optimization levels which pyc can be compiled for (1-3) [0][1].
I think (untested) the current rsplit will fail on an optimized file pattern and we can be a bit more future-proof so we don't accidentally run into this same problem again?
The thing is, we currently do not generate "optimised" bytecode, so I have no way of:
1. knowing how the files are named
2. test any change.
Quoted 1 line
Maybe we can leverage
source_from_cache[2] and compare the file names?
Ah, good idea if it can work out-of-tree as we install for the target. I'll look into that.
Quoted 3 lines
It's probably overkill, but I think this is one of those easily overlooked things where if we did add support for optimization levels that we'd miss fixing the code here to account for it.
Assuming this function handles all the magic, it is better than open-coding the path resolution...
Regards,
Yann E. MORIN.
PS. Sorry for the delay in my reply, I'm just back from my holiday break.
Signature · 23 lines
____________ .-----------------.--------------------: _ :------------------. | Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON | | | Software Designer | _/ - /' | \ / CAMPAIGN | | +33 638.411.245 '--------------------: (_ `--, | X AGAINST | | yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL | '--------------------------------------:______/_____:------------------' ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc files
Vincent, All,
On 2026-09-14 11:27 +0200, MORIN Yann INNOV/IT-S spake thusly:
Quoted 1 line
On 2026-08-19 13:44 +0000, Vincent Fazio spake thusly:
[--SNIP--]
Quoted 3 lines
Quoted 1 line
Maybe we can leverage
source_from_cache[2] and compare the file names?Ah, good idea if it can work out-of-tree as we install for the target. I'll look into that.
Oh, wait, wait... That's the other way around: we do have the path of the original .py file, and we are looking for the path of the pre-compiled file!
So I gave it a try with cache_from_source() [0] instead, and it kinda of works...
"Kinda", because it only works when we do have both the source and the pre-compiled files together (BR2_PACKAGE_PYTHON3_PY_PYC), but fails when we only have the precompiled files (BR2_PACKAGE_PYTHON3_PYC_ONLY). That's because cache_from_source() only work on the path string, and does not check files.
But that is not an issue in practice:
1. BR2_PACKAGE_PYTHON3_PY_ONLY: we only have .py files, so when we try to look at the cprfe-copmpiled files, it will not be found, and thus ignored (as it is today already)
2. BR2_PACKAGE_PYTHON3_PYC_ONLY: we'll only have the .pyc file in the pycache directory, and the .py file will be ignored (as it is today already).
3. BR2_PACKAGE_PYTHON3_PY_PYC: both will be found, and we'll properly account for both (unlike today)
So, that's what I'm going to go with, using cache_from_source(), rather than with my open-coding the split. Thanks for the hint! 👍
However, I have not seen a way to convert from a source file path to an *optimised* precompiled pyc file. So I still believe this should be addressed when (if!) we eventually add squpport for optimised precompiled pyc files. If we forget about size-stat then, that's not a
problem: the interested parties will have to fixup afterward (like I'm currently doing anyway ;-) ).
Regards,
Yann E. MORIN.
Signature · 23 lines
____________ .-----------------.--------------------: _ :------------------. | Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON | | | Software Designer | _/ - /' | \ / CAMPAIGN | | +33 638.411.245 '--------------------: (_ `--, | X AGAINST | | yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL | '--------------------------------------:______/_____:------------------' ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [External] - Re: [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc files
Yann,
Quoted 11 lines
-----Original Message-----
From: buildroot <[email protected]> On Behalf Of [email protected]
Sent: Monday, September 14, 2026 7:00 AM
To: Vincent Fazio <[email protected]>
Cc: [email protected]
Subject: [External] - Re: [Buildroot] [PATCH 2/2] support/scripts/size-stats:
properly assign pre-compiled .pyc filesVincent, All,
Quoted 7 lines
On 2026-09-14 11:27 +0200, MORIN Yann INNOV/IT-S spake thusly:
So, that's what I'm going to go with, using
cache_from_source(), rather than with my open-coding the split. Thanks for the hint! 👍However, I have not seen a way to convert from a source file path to an *optimised* precompiled pyc file.
cache_from_source should do this:
```
import importlib.util
importlib.util.cache_from_source("/a/b/c/d.py")
'/a/b/c/__pycache__/d.cpython-314.pyc'
importlib.util.cache_from_source("/a/b/c/d.py", optimization=1)
'/a/b/c/__pycache__/d.cpython-314.opt-1.pyc'
importlib.util.cache_from_source("/a/b/c/d.py", optimization=2)
'/a/b/c/__pycache__/d.cpython-314.opt-2.pyc'
```_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello All,
On 2026-08-10 15:27 +0200, [email protected] spake thusly:
Quoted 2 lines
This small two-patch series properly associates .pyc files to the corresponding package.
Even though rc1 has been tagged, I believe this is still material for master, as this is a fix.
Any feedback on the patches, in case I need to rework something before the release? ;-)
Regards,
Yann E. MORIN.
Signature · 23 lines
____________ .-----------------.--------------------: _ :------------------. | Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON | | | Software Designer | _/ - /' | \ / CAMPAIGN | | +33 638.411.245 '--------------------: (_ `--, | X AGAINST | | yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL | '--------------------------------------:______/_____:------------------' ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot