Skip to contentSkip to conversation

package/micropython: make use of recent micropython-lib

PATCH Applied Abilio Marques 2 participants 3 messages Jan 29, 2024

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 package/micropython: make use of recent micropython-lib

Until now, micropython-lib was a package that installed v1.9.3, which is more than 6 years old. This was acceptable since micropython never made any other official release of the library until v1.20.

Meanwhile, the libraries underwent a reorganization, and they are now available in a directory structure that cannot be copied directly into the target. This is my theory on why v1.9.3 is still present in the current day buildroot (which comes with micropython v1.22).

This commit introduces an auxiliary script to collect those libraries and reorder them into a structure that can then be copied into /usr/lib/micropython. The script utilizes a module from the tools directory of the micropython repo.

As part of the changes made by the micropython project, the libraries are now released together with the interpreter. They are cloned as a submodule into the lib/micropython-lib directory.

These are the 2 main reasons why the old buildroot micropython-lib package is being removed.

With this commit, micropython-lib is installed (optionally) as part of micropython. The original config variable name was retained as it fits with the micropython package 'namespace", and thus this is backward compatible and no legacy handling is needed.

I tried to keep it simple. It makes use of existing micropython tools (used to process manifests) to discover the list of packages available in micropython-lib. My hope is that by relying on them, any future changes in directory structure will be covered by the official "manifestfile.py" tool.

This commit also ensures that the libraries in micropython-lib will be updated together with newer versions of micropython.

---
 DEVELOPERS                                    |  1 -
 package/Config.in                             |  1 -
 package/micropython-lib/Config.in             |  8 --
 package/micropython-lib/micropython-lib.hash  |  3 -
 package/micropython-lib/micropython-lib.mk    | 18 -----
 package/micropython/Config.in                 |  8 ++
 .../micropython/collect_micropython_lib.py    | 79 +++++++++++++++++++
 package/micropython/micropython.mk            | 17 ++++
 8 files changed, 104 insertions(+), 31 deletions(-)
 delete mode 100644 package/micropython-lib/Config.in
 delete mode 100644 package/micropython-lib/micropython-lib.hash
 delete mode 100644 package/micropython-lib/micropython-lib.mk
 create mode 100755 package/micropython/collect_micropython_lib.py
DEVELOPERS +0 -1
index 9528837dd0..cc9bc5b5cc 100644
@@ -572,7 +572,6 @@ F:	package/coremark/
 F:	package/coremark-pro/
 F:	package/gstreamer1/gst1-shark/
 F:	package/micropython/
-F:	package/micropython-lib/
 F:	package/syslog-ng/
 
 N:	Christian Kellermann <[email protected]>
package/Config.in +0 -1
index 5b8b15fa54..426bd7d090 100644
@@ -798,7 +798,6 @@ menu "Lua libraries/modules"
 endmenu
 endif
 	source "package/micropython/Config.in"
-	source "package/micropython-lib/Config.in"
 	source "package/moarvm/Config.in"
 	source "package/mono/Config.in"
 if BR2_PACKAGE_MONO
package/micropython-lib/Config.in deleted +0 -8
deleted file mode 100644
index 76557b220b..0000000000
@@ -1,8 +0,0 @@
-config BR2_PACKAGE_MICROPYTHON_LIB
-	bool "micropython-lib"
-	depends on BR2_PACKAGE_MICROPYTHON
-	select BR2_PACKAGE_PCRE # runtime
-	help
-	  Core Python libraries ported to MicroPython.
-
-	  http://micropython.org
package/micropython-lib/micropython-lib.hash deleted +0 -3
deleted file mode 100644
index cbdda23844..0000000000
@@ -1,3 +0,0 @@
-# Locally computed
-sha256  66e15380eb109613263beb6825b8eecb9191088270c1a59e8c7d922dd57183c7  micropython-lib-1.9.3.tar.gz
-sha256  baed4196a4310c576c2010f0a49f987a49e63856df7cd45af11cb3571df4bf74  LICENSE
package/micropython-lib/micropython-lib.mk deleted +0 -18
deleted file mode 100644
index 78ac0d3b35..0000000000
@@ -1,18 +0,0 @@
-################################################################################
-#
-# micropython-lib
-#
-################################################################################
-
-MICROPYTHON_LIB_VERSION = 1.9.3
-MICROPYTHON_LIB_SITE = $(call github,micropython,micropython-lib,v$(MICROPYTHON_LIB_VERSION))
-MICROPYTHON_LIB_LICENSE = Python-2.0 (some modules), MIT (everything else)
-MICROPYTHON_LIB_LICENSE_FILES = LICENSE
-
-define MICROPYTHON_LIB_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
-		PREFIX=$(TARGET_DIR)/usr/lib/micropython \
-		install
-endef
-
-$(eval $(generic-package))
package/micropython/Config.in +8 -0
index 30161c8b70..966e4b958e 100644
@@ -9,5 +9,13 @@ config BR2_PACKAGE_MICROPYTHON
 
 	  http://micropython.org
 
+config BR2_PACKAGE_MICROPYTHON_LIB
+	bool "micropython-lib"
+	default y
+	depends on BR2_PACKAGE_MICROPYTHON
+	select BR2_PACKAGE_PCRE # runtime
+	help
+	  Core Python libraries ported to MicroPython.
+
 comment "micropython needs a toolchain w/ threads, dynamic library"
 	depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
package/micropython/collect_micropython_lib.py new file +79 -0

79 lines not on this page · raw message

package/micropython/micropython.mk +17 -0
index 37c148da94..5a0c5616a5 100644
@@ -57,4 +57,21 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
 		install
 endef
 
+ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
+define MICROPYTHON_COLLECT_LIBS
+	$(EXTRA_ENV) PYTHONPATH=$(@D)/tools:/$PYTHONPATH \
+		package/micropython/collect_micropython_lib.py \
+		$(@D) $(@D)/.built_pylib
+endef
+
+define MICROPYTHON_INSTALL_LIBS
+	$(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
+	cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
+endef
+
+MICROPYTHON_POST_BUILD_HOOKS += MICROPYTHON_COLLECT_LIBS
+MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
+endif
+
+
 $(eval $(generic-package))
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Abilio, All,

Thanks for this new iteration!

On 2024-01-28 18:29 -0800, Abilio Marques spake thusly:

Quoted 7 lines

Until now, micropython-lib was a package that installed v1.9.3, which is more than 6 years old. This was acceptable since micropython never made any other official release of the library until v1.20.

Meanwhile, the libraries underwent a reorganization, and they are now available in a directory structure that cannot be copied directly into the target. This is my theory on why v1.9.3 is still present in the

We try to avoid singular first-person messages in the commit log, and prefer the more neutral plural first-person "we", or the passive tense. Here, I'd rephrase it as;

This might explain why v1.9.3 is still...

Quoted 19 lines

current day buildroot (which comes with micropython v1.22).

This commit introduces an auxiliary script to collect those libraries and reorder them into a structure that can then be copied into /usr/lib/micropython. The script utilizes a module from the tools directory of the micropython repo.

As part of the changes made by the micropython project, the libraries are now released together with the interpreter. They are cloned as a submodule into the lib/micropython-lib directory.

These are the 2 main reasons why the old buildroot micropython-lib package is being removed.

With this commit, micropython-lib is installed (optionally) as part of micropython. The original config variable name was retained as it fits with the micropython package 'namespace", and thus this is backward compatible and no legacy handling is needed.

Quoted 5 lines

I tried to keep it simple. It makes use of existing micropython tools (used to process manifests) to discover the list of packages available in micropython-lib. My hope is that by relying on them, any future changes in directory structure will be covered by the official "manifestfile.py" tool.

Ditto, no "I" or "my". No need to resend just ofe of the phrasing, it can be fixed when applying.

One thing that again got me down the host-micropython rabbit-hole, is that we don't need host-micropython at all, because the manifestfile module is actual CPython code, not micropython code, so we really, like really-really, want to write that script for CPython.

[--SNIP--]

package/micropython/micropython.mk define MICROPYTHON_INSTALL_TARGET_CMDS · line 62

Quoted 11 lines
package/micropython/micropython.mk +3 -0
index 37c148da94..5a0c5616a5 100644
@@ -57,4 +57,21 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
 		install
 endef
 
+ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
+define MICROPYTHON_COLLECT_LIBS
+	$(EXTRA_ENV) PYTHONPATH=$(@D)/tools:/$PYTHONPATH \

There is no need to complement PYTHON path with $PYTHONPATH. Indeed, PYTHONPATH only *augments* the search path, it does not replace the standard path, so we should just need:

    $(EXTRA_ENV) PYTHONPATH=$(@D)/tools \
    package/micropython/collect_micropython_lib.py \
        $(@D) $(@D)/.built_pylib

No need to resend, I'll tinker with it tonight.

Thanks!

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw
In reply to Abilio Marques

Abilio, All,

On 2024-01-28 18:29 -0800, Abilio Marques spake thusly:

Quoted 32 lines

Until now, micropython-lib was a package that installed v1.9.3, which is more than 6 years old. This was acceptable since micropython never made any other official release of the library until v1.20.

Meanwhile, the libraries underwent a reorganization, and they are now available in a directory structure that cannot be copied directly into the target. This is my theory on why v1.9.3 is still present in the current day buildroot (which comes with micropython v1.22).

This commit introduces an auxiliary script to collect those libraries and reorder them into a structure that can then be copied into /usr/lib/micropython. The script utilizes a module from the tools directory of the micropython repo.

As part of the changes made by the micropython project, the libraries are now released together with the interpreter. They are cloned as a submodule into the lib/micropython-lib directory.

These are the 2 main reasons why the old buildroot micropython-lib package is being removed.

With this commit, micropython-lib is installed (optionally) as part of micropython. The original config variable name was retained as it fits with the micropython package 'namespace", and thus this is backward compatible and no legacy handling is needed.

I tried to keep it simple. It makes use of existing micropython tools (used to process manifests) to discover the list of packages available in micropython-lib. My hope is that by relying on them, any future changes in directory structure will be covered by the official "manifestfile.py" tool.

Quoted 4 lines

This commit also ensures that the libraries in micropython-lib will be updated together with newer versions of micropython.

I have slightly reworded and somewhat erorganmised th ecommit log in a more logical sequence, but it was already very good to begin with!

[--SNIP--]

package/micropython/Config.in config BR2_PACKAGE_MICROPYTHON · line 14

Quoted 11 lines
package/micropython/Config.in +3 -0
index 30161c8b70..966e4b958e 100644
@@ -9,5 +9,13 @@ config BR2_PACKAGE_MICROPYTHON
 
 	  http://micropython.org
 
+config BR2_PACKAGE_MICROPYTHON_LIB
+	bool "micropython-lib"
+	default y

There is no reason to 'default y' here, as the previously separate micropython-lib did not. The new ones are no more required than the old ones, and we really need default y' to hae a good rationale. If you think there is one, then do not hesitate to send a follow up patch that explains why switching to 'default y' is needed.

package/micropython/Config.in config BR2_PACKAGE_MICROPYTHON · line 15

Quoted 1 line
package/micropython/Config.in +1 -0
+	depends on BR2_PACKAGE_MICROPYTHON

Even for a single sub-options, we prefer an enclosing if-block, so I converted the dependency to that.

[--SNIP--]

package/micropython/micropython.mk define MICROPYTHON_INSTALL_TARGET_CMDS · line 62

Quoted 11 lines
package/micropython/micropython.mk +3 -0
index 37c148da94..5a0c5616a5 100644
@@ -57,4 +57,21 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
 		install
 endef
 
+ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
+define MICROPYTHON_COLLECT_LIBS
+	$(EXTRA_ENV) PYTHONPATH=$(@D)/tools:/$PYTHONPATH \

And I have indeed dropped the latter part of PYTHONPATH, because it is not useful and is even incorrect.

Quoted 4 lines
package/micropython/micropython.mk +4 -0
+		package/micropython/collect_micropython_lib.py \
+		$(@D) $(@D)/.built_pylib
+endef
+

package/micropython/micropython.mk define MICROPYTHON_INSTALL_TARGET_CMDS · line 76

Quoted 10 lines
package/micropython/micropython.mk +10 -0
+define MICROPYTHON_INSTALL_LIBS
+	$(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
+	cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
+endef
+
+MICROPYTHON_POST_BUILD_HOOKS += MICROPYTHON_COLLECT_LIBS
+MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
+endif
+
+
    $ ./utils/docker-run make check-package
    package/micropython/micropython.mk:76: consecutive empty lines

So I've fixed that too. So here are the few minoor changes I made:

- use if-block in Config.in
- simplify PYTHONPATH
- fix check-package
- reword and reorder parts of the commit log

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 8 lines
package/micropython/micropython.mk +0 -0
 $(eval $(generic-package))
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot