package/micropython: add support for manifest.py in the configuration
PATCH 1/1 package/micropython: add support for manifest.py in the configuration
Micropython can embed packages an modules as frozen bytecode. What code gets built this way can be defined by means of a "manifest.py" file.
This commit exposes the variable FROZEN_MANIFEST to Buildroot users through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST.
Please check Micropython's documentation for more information:
https://docs.micropython.org/en/latest/reference/manifest.html
--- package/micropython/Config.in | 8 ++++++++ package/micropython/micropython.mk | 5 +++++ 2 files changed, 13 insertions(+)
package/micropython/Config.in +8 -0
@@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB help Core Python libraries ported to MicroPython. +config BR2_PACKAGE_MICROPYTHON_MANIFEST + string "Path to a manifest.py file" + help + MicroPython allows Python code to be “frozen” as bytecode + into its binary, as an alternative to loading code from + the filesystem. See MicroPython's documentation for more + information. + endif # BR2_PACKAGE_MICROPYTHON comment "micropython needs a toolchain w/ threads, dynamic library"
package/micropython/micropython.mk +5 -0
@@ -43,6 +43,11 @@ else MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0 endif +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"") +MICROPYTHON_MAKE_OPTS += \ + FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST) +endif + define MICROPYTHON_BUILD_CMDS $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Abilio, All,
On 2024-02-03 22:26 -0800, Abilio Marques spake thusly:
Quoted 8 lines
Micropython can embed packages an modules as frozen bytecode. What code gets built this way can be defined by means of a "
manifest.py" file.This commit exposes the variable
FROZEN_MANIFESTto Buildroot users through a new variable calledBR2_PACKAGE_MICROPYTHON_MANIFEST.Please check Micropython's documentation for more information:
https://docs.micropython.org/en/latest/reference/manifest.html
Minor nit: this blurb should have been in the help text of the new option, and I could have done that when applying, but I have a much more involved question, see below.
Quoted 30 lines
Signed-off-by Abilio Marques <[email protected]>--- package/micropython/Config.in | 8 ++++++++ package/micropython/micropython.mk | 5 +++++ 2 files changed, 13 insertions(+)package/micropython/Config.in +8
-0@@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB help Core Python libraries ported to MicroPython. +config BR2_PACKAGE_MICROPYTHON_MANIFEST + string "Path to a manifest.py file" + help + MicroPython allows Python code to be “frozen” as bytecode + into its binary, as an alternative to loading code from + the filesystem. See MicroPython's documentation for more + information. + endif # BR2_PACKAGE_MICROPYTHON comment "micropython needs a toolchain w/ threads, dynamic library"package/micropython/micropython.mk +0
-0
package/micropython/micropython.mk else · line 49
Quoted 7 lines
package/micropython/micropython.mk +4
-0@@ -43,6 +43,11 @@ else MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0 endif +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"") +MICROPYTHON_MAKE_OPTS += \ + FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST) +endif
So, as I understand this, micropython will grab the files (from modules, files, etc...) listed from the manifest and bundle them in the micropython executable. It seems the paths in the manifest can be either absolute, or relative; in the latter case, it not documented what they would be relative to; additionally there are placeholders (see below) that can be used but are absolute paths.
So this has a few implications:
- the files must be available before micropython is built, so some dependency order is needed to ensure that: if files are provided in a package, then we need a dependency and it is going to be a little bit difficult to do and we need a way to address that;
- the paths can't be hard-coded to absolute paths in the manifest file
itself, because we don't know beforehand where the buildroot build
directory will be, so we need a way to shoehorn $(TOPDIR) or some
such variable in the manifest;- absolute paths must be fixed to accommodate the per-package option, because those paths change for eaach package.
From the micropython doc, that you pointed above, there are a few placeholders that will be replaced in the manifest, but I don't think we can reuse those:
Any paths used in manifest files can include the following variables. These all resolve to absolute paths.
* $(MPY_DIR) – path to the micropython repo.
* $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to use require().
* $(PORT_DIR) – path to the current port (e.g. ports/stm32)
* $(BOARD_DIR) – path to the current board (e.g. ports/stm32/boards/PYBV11)
So we need a bit more information how this feature is to be used, and an example (or a few) would probably help a lot in this respect. We also have a runtime test for micropython, in:
support/testing/tests/package/test_micropython.py
It would be nice to extend that runtime test as well.
Regards,
Yann E. MORIN.
Quoted 10 lines
package/micropython/micropython.mk +0
-0define MICROPYTHON_BUILD_CMDS $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Yann,
One of the uses of manifest.py is to specify which modules of micropython-lib should be frozen within the binary. For those modules you don't need to include the path. e.g.,
package('os')
That's the application I'm going for. I know it seems limited but it's really useful, and probably welcomed by other people using Micropython in the next release of Buildroot.
I have ideas on how to solve the path and dependency order problem while still using the "official" manifest.py concept. The biggest challenge is that currently there are no other 3rd party Micropython modules available for Buildroot, so that makes it all very theoretical. All solutions require a bigger amount of work than the one needed for this patch. Also, I would like to discuss them before actually presenting a patch that allows 3rd party modules to be frozen.
I always try to go for an incremental approach, where I get the bigger bang for the buck. I believe that allowing people to freeze the official Micropython modules is already a big step forward. But at the same time, I'm new to the Buildroot project, so please advice on the approach.
Kind regards,
Abilio
Quoted 124 lines · Yann E. MORIN
Abilio, All,
On 2024-02-03 22:26 -0800, Abilio Marques spake thusly:
Quoted 8 lines
Micropython can embed packages an modules as frozen bytecode. What code gets built this way can be defined by means of a "
manifest.py" file.This commit exposes the variable
FROZEN_MANIFESTto Buildroot users through a new variable calledBR2_PACKAGE_MICROPYTHON_MANIFEST.Please check Micropython's documentation for more information:
https://docs.micropython.org/en/latest/reference/manifest.htmlMinor nit: this blurb should have been in the help text of the new option, and I could have done that when applying, but I have a much more involved question, see below.
Review
Quoted 7 lines
Signed-off-by Abilio Marques <[email protected]>--- package/micropython/Config.in | 8 ++++++++ package/micropython/micropython.mk | 5 +++++ 2 files changed, 13 insertions(+)(quoted diff) +0
-0b/package/micropython/Config.in
package/micropython/
Quoted 19 lines
index
26a00baab0..52717d815a100644package/micropython/Config.in +8
-0@@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB help Core Python libraries ported to MicroPython. +config BR2_PACKAGE_MICROPYTHON_MANIFEST + string "Path to a manifest.py file" + help + MicroPython allows Python code to be “frozen” as bytecode + into its binary, as an alternative to loading code from + the filesystem. See MicroPython's documentation for more + information. + endif # BR2_PACKAGE_MICROPYTHON comment "micropython needs a toolchain w/ threads, dynamic library"package/micropython/ renamed from package/micropython/micropython.mk +0
-0micropython.mk
package/micropython/micropython.mk else · line 49
Quoted 11 lines
index
125a0edcfb..5a2c136547100644package/micropython/micropython.mk +4
-0@@ -43,6 +43,11 @@ else MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0 endif +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"") +MICROPYTHON_MAKE_OPTS += \ + FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST) +endifSo, as I understand this, micropython will grab the files (from modules, files, etc...) listed from the manifest and bundle them in the micropython executable. It seems the paths in the manifest can be either absolute, or relative; in the latter case, it not documented what they would be relative to; additionally there are placeholders (see below) that can be used but are absolute paths.
So this has a few implications:
- the files must be available before micropython is built, so some dependency order is needed to ensure that: if files are provided in a package, then we need a dependency and it is going to be a little bit difficult to do and we need a way to address that;
- the paths can't be hard-coded to absolute paths in the manifest file
itself, because we don't know beforehand where the buildroot build directory will be, so we need a way to shoehorn $(TOPDIR) or some such variable in the manifest;- absolute paths must be fixed to accommodate the per-package option, because those paths change for eaach package.
From the micropython doc, that you pointed above, there are a few placeholders that will be replaced in the manifest, but I don't think we can reuse those:
Any paths used in manifest files can include the following variables. These all resolve to absolute paths.
* $(MPY_DIR) – path to the micropython repo.
* $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to
userequire().
* $(PORT_DIR) – path to the current port (e.g. ports/stm32)
* $(BOARD_DIR) – path to the current board (e.g.
ports/stm32/boards/PYBV11)So we need a bit more information how this feature is to be used, and an example (or a few) would probably help a lot in this respect. We also have a runtime test for micropython, in:
support/testing/tests/package/test_micropython.pyIt would be nice to extend that runtime test as well.
Regards,
Yann E. MORIN.Quoted 10 lines
package/micropython/micropython.mk +0
-1define MICROPYTHON_BUILD_CMDS $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \ --2.43.0
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
Quoted 2 lines
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Abilio, All,
[Please, don't top-post, but reply in-line]
On 2024-02-11 09:15 -0800, Abilio Marques spake thusly:
Quoted 7 lines
One of the uses of
manifest.pyis to specify which modules of micropython-lib should be frozen within the binary. For those modules you don't need to include the path. e.g.,
package('os')That's the application I'm going for. I know it seems limited but it's really useful,
OK, so maybe this can be explained in the help text of the new option, like:
Note: in Buildroot, only modules provided with micropython-lib can be frozen with a manifest; freezing arbitrary files is not supported.
along with a little blurb in the commit log, statng something like;
We only support freezing of modules from micropython-lib. Freezing arbitrary modules would require some handling of absoulte paths, and a guarantee that the files be available before micropython is built, which is considered a bit too complex; an interested party can provide such support in the future.
And thus, would it be possible to sanity-check the manifest to ensure that it indeed only references modules from icropython-lib, so that people do not get the impression it works when in fact it does not?
Like, grep that only require(...) are used?
(Note: you mentioned package(...) but that is not limited to files from micropython-lib; require(...) however is.)
Quoted 2 lines
and probably welcomed by other people using Micropython in the next release of Buildroot.
One question I was wondering about: if a module is frozen in the binary, then it is no longer needed in the filesystem, right? If so, should we have a way to remove them?
Quoted 7 lines
I have ideas on how to solve the path and dependency order problem while still using the "official"
manifest.pyconcept. The biggest challenge is that currently there are no other 3rd party Micropython modules available for Buildroot, so that makes it all very theoretical. All solutions require a bigger amount of work than the one needed for this patch. Also, I would like to discuss them before actually presenting a patch that allows 3rd party modules to be frozen.
Indeed, without a few actual examples, it's going to be difficult to see a common pattern and abstract that away. Are there any pulicly available?
Quoted 4 lines
I always try to go for an incremental approach, where I get the bigger bang for the buck. I believe that allowing people to freeze the official Micropython modules is already a big step forward. But at the same time, I'm new to the Buildroot project, so please advice on the approach.
The incremental path is totally OK; I even prefer it. Of course, any limitation (such as those we are discussing) should be explained in the commit log.
Given all the feedback in this thread, can you respin a v2 taking the comments into account?
As for your ideas for lifting those limitations, you can just explain them in a reply in this thread. Usually, a patchset doing the job is also a good first step to start the dicsussion.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 29 lines · Yann E. MORIN
Abilio, All,
[Please, don't top-post, but reply in-line]
On 2024-02-11 09:15 -0800, Abilio Marques spake thusly:
Quoted 7 lines
One of the uses of
manifest.pyis to specify which modules of micropython-lib should be frozen within the binary. For those modules you don't need to include the path. e.g.,
package('os')That's the application I'm going for. I know it seems limited but it's really useful,
OK, so maybe this can be explained in the help text of the new option, like:
Note: in Buildroot, only modules provided with micropython-lib can be frozen with a manifest; freezing arbitrary files is not supported.
along with a little blurb in the commit log, statng something like;
We only support freezing of modules from micropython-lib. Freezing arbitrary modules would require some handling of absoulte paths, and a guarantee that the files be available before micropython is built, which is considered a bit too complex; an interested party can provide such support in the future.
Quoted 5 lines
And thus, would it be possible to sanity-check the manifest to ensure that it indeed only references modules from icropython-lib, so that people do not get the impression it works when in fact it does not?
Like, grep that only require(...) are used?
Not sure how to do it correctly. The manifest is a regular python file, and I'm not sure how to do it in a way that does the correct the thing. e.g., the user might
want to add comment lines or an include to a second part of the manifest.
Quoted 10 lines
(Note: you mentioned package(...) but that is not limited to files from micropython-lib; require(...) however is.)
Quoted 2 lines
and probably welcomed by other people using Micropython in the next release of Buildroot.
One question I was wondering about: if a module is frozen in the binary, then it is no longer needed in the filesystem, right? If so, should we have a way to remove them?
Good idea. Do you mind if I add this in a separate patch?
Quoted 18 lines
Quoted 8 lines
I have ideas on how to solve the path and dependency order problem while still using the "official"
manifest.pyconcept. The biggest challenge is that currently there are no other 3rd party Micropython modules available for Buildroot, so that makes it all very theoretical. All solutions require a bigger amount of work than the one needed for this patch. Also, I would like to discuss them before actually presenting a patch that allows 3rd party modules to be frozen.Indeed, without a few actual examples, it's going to be difficult to see a common pattern and abstract that away. Are there any pulicly available?
Quoted 4 lines
I always try to go for an incremental approach, where I get the bigger bang for the buck. I believe that allowing people to freeze the official Micropython modules is already a big step forward. But at the same time, I'm new to the Buildroot project, so please advice on the approach.
Quoted 6 lines
The incremental path is totally OK; I even prefer it. Of course, any limitation (such as those we are discussing) should be explained in the commit log.
Given all the feedback in this thread, can you respin a v2 taking the comments into account?
I did, but just found out that I submitted a patch with a whitespace problem, in this line
"can be frozen with a manifest; freezing arbitrary files is"
Could you please correct it?
Quoted 4 lines
As for your ideas for lifting those limitations, you can just explain them in a reply in this thread. Usually, a patchset doing the job is also a good first step to start the dicsussion.
Will try a few things and submit one.
Quoted 11 lines
Regards,
Yann E. MORIN.--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot