Skip to contentSkip to conversation

package/python-babel: add purge locale data support

RFC In review Peter Seiderer 3 participants 6 messages Apr 18, 2023 – Aug 1, 2023

lore
Share & tools
lore
#0 lore raw

RFC 1/1 package/python-babel: add purge locale data support

Add purge loacle data support (inspired by BR2_ENABLE_LOCALE_PURGE handling in Makefile) to reduce target space allocation by the python-babel package, e.g. from 32MB to 24K for target/usr/lib/python3.11/site-packages/babel/locale-data with BR2_ENABLE_LOCALE_WHITELIST="C en_US de_DE".

---
 package/python-babel/python-babel.mk | 12 ++++++++++++
 1 file changed, 12 insertions(+)
package/python-babel/python-babel.mk +12 -0
index 890a8b02b9..ea383e9c05 100644
@@ -12,5 +12,17 @@ PYTHON_BABEL_LICENSE = BSD-3-Clause
 PYTHON_BABEL_LICENSE_FILES = LICENSE
 HOST_PYTHON_BABEL_DEPENDENCIES = host-python-pytz
 
+# purge locale data (if enabled), keep special en_US_POSIX data by default
+ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
+define PYTHON_BABEL_CLEANUP_LOCALE
+	for i in `ls $(TARGET_DIR)/usr/lib/python3.11/site-packages/babel/locale-data/*.dat`; \
+	do \
+		i_base=`basename "$$i" .dat`; \
+		echo "$(BR2_ENABLE_LOCALE_WHITELIST) en_US_POSIX" | grep -qw "$$i_base" || rm "$$i"; \
+	done
+endef
+endif
+PYTHON_BABEL_POST_INSTALL_TARGET_HOOKS += PYTHON_BABEL_CLEANUP_LOCALE
+
 $(eval $(python-package))
 $(eval $(host-python-package))
Signature · 6 lines
2.40.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Peter, All,

On 2023-04-18 23:32 +0200, Peter Seiderer spake thusly:

Quoted 16 lines

Add purge loacle data support (inspired by BR2_ENABLE_LOCALE_PURGE handling in Makefile) to reduce target space allocation by the python-babel package, e.g. from 32MB to 24K for target/usr/lib/python3.11/site-packages/babel/locale-data with BR2_ENABLE_LOCALE_WHITELIST="C en_US de_DE".

---
 package/python-babel/python-babel.mk | 12 ++++++++++++
 1 file changed, 12 insertions(+)
package/python-babel/python-babel.mk +0 -0
index 890a8b02b9..ea383e9c05 100644

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 21

Quoted 10 lines
package/python-babel/python-babel.mk +7 -0
@@ -12,5 +12,17 @@ PYTHON_BABEL_LICENSE = BSD-3-Clause
 PYTHON_BABEL_LICENSE_FILES = LICENSE
 HOST_PYTHON_BABEL_DEPENDENCIES = host-python-pytz
 
+# purge locale data (if enabled), keep special en_US_POSIX data by default
+ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
+define PYTHON_BABEL_CLEANUP_LOCALE
+	for i in `ls $(TARGET_DIR)/usr/lib/python3.11/site-packages/babel/locale-data/*.dat`; \
+	do \
+		i_base=`basename "$$i" .dat`; \
+		echo "$(BR2_ENABLE_LOCALE_WHITELIST) en_US_POSIX" | grep -qw "$$i_base" || rm "$$i"; \

Why do you force keeping the en_US_POSIX locale specifically?

If there is the need that at least one locale be present, then can we keep a more generic one, like 'C', or 'POSIX', or whatever is not specific to a country?

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 25

Quoted 4 lines
package/python-babel/python-babel.mk +4 -0
+	done
+endef
+endif
+PYTHON_BABEL_POST_INSTALL_TARGET_HOOKS += PYTHON_BABEL_CLEANUP_LOCALE

Like for the toolchain locale purge, it should be a target-fnalize hook;
PYTHON_BABEL_TARGET_FINALIZE_HOOKS

(Note that the systemd locale purge is done as a ROOTFS_PRE_CMD_HOOKS, but that is wrong, IMNSHO, and it too should have been a target finalize hook.)

Regards,
Yann E. MORIN.

Quoted 9 lines
package/python-babel/python-babel.mk +0 -0
 $(eval $(python-package))
 $(eval $(host-python-package))
Signature · 6 lines
2.40.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Hello Yann,

Quoted 22 lines · Yann E. MORIN

Peter, All,

On 2023-04-18 23:32 +0200, Peter Seiderer spake thusly:

Quoted 19 lines

Add purge loacle data support (inspired by BR2_ENABLE_LOCALE_PURGE handling in Makefile) to reduce target space allocation by the python-babel package, e.g. from 32MB to 24K for target/usr/lib/python3.11/site-packages/babel/locale-data with BR2_ENABLE_LOCALE_WHITELIST="C en_US de_DE".

---
 package/python-babel/python-babel.mk | 12 ++++++++++++
 1 file changed, 12 insertions(+)
package/python-babel/python-babel.mk +0 -0
index 890a8b02b9..ea383e9c05 100644
@@ -12,5 +12,17 @@ PYTHON_BABEL_LICENSE = BSD-3-Clause
 PYTHON_BABEL_LICENSE_FILES = LICENSE
 HOST_PYTHON_BABEL_DEPENDENCIES = host-python-pytz
 
Quoted 9 lines

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 21

Quoted 7 lines
package/python-babel/python-babel.mk +7 -0
+# purge locale data (if enabled), keep special en_US_POSIX data by default
+ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
+define PYTHON_BABEL_CLEANUP_LOCALE
+	for i in `ls $(TARGET_DIR)/usr/lib/python3.11/site-packages/babel/locale-data/*.dat`; \
+	do \
+		i_base=`basename "$$i" .dat`; \
+		echo "$(BR2_ENABLE_LOCALE_WHITELIST) en_US_POSIX" | grep -qw "$$i_base" || rm "$$i"; \

Why do you force keeping the en_US_POSIX locale specifically?

Because it is the python-babel default, see [1]...

Quoted 4 lines

If there is the need that at least one locale be present, then can we keep a more generic one, like 'C', or 'POSIX', or whatever is not specific to a country?

See above...

Quoted 8 lines

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 25

Quoted 5 lines
package/python-babel/python-babel.mk +4 -0
+	done
+endef
+endif
+PYTHON_BABEL_POST_INSTALL_TARGET_HOOKS += PYTHON_BABEL_CLEANUP_LOCALE

Like for the toolchain locale purge, it should be a target-fnalize hook;
PYTHON_BABEL_TARGET_FINALIZE_HOOKS

Will fix it...

Thanks for review!

Regards,
Peter

[1] https://github.com/python-babel/babel/blob/0ce196fccc024b1a65453ba6519954ada1dab6cb/babel/core.py#L1076

Quoted 18 lines

(Note that the systemd locale purge is done as a ROOTFS_PRE_CMD_HOOKS, but that is wrong, IMNSHO, and it too should have been a target finalize hook.)

Regards,
Yann E. MORIN.

Quoted 9 lines
package/python-babel/python-babel.mk +0 -1
 $(eval $(python-package))
 $(eval $(host-python-package))
--

2.40.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#3 lore raw

Peter, All,

On 2023-05-15 22:49 +0200, Peter Seiderer spake thusly:

Quoted 7 lines · Yann E. MORIN
Quoted 6 lines

On 2023-04-18 23:32 +0200, Peter Seiderer spake thusly:

Quoted 5 lines

Add purge loacle data support (inspired by BR2_ENABLE_LOCALE_PURGE handling in Makefile) to reduce target space allocation by the python-babel package, e.g. from 32MB to 24K for target/usr/lib/python3.11/site-packages/babel/locale-data with BR2_ENABLE_LOCALE_WHITELIST="C en_US de_DE".

[--SNIP--]

Quoted 3 lines
Quoted 2 lines

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 21

Quoted 1 line
package/python-babel/python-babel.mk +1 -0
+		echo "$(BR2_ENABLE_LOCALE_WHITELIST) en_US_POSIX" | grep -qw "$$i_base" || rm "$$i"; \

Why do you force keeping the en_US_POSIX locale specifically?

Because it is the python-babel default, see [1]...

OK, Thanks! Say so explicitly in the commit log when you respin.

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw
In reply to Peter Seiderer
Quoted 5 lines · On Tue, 18 Apr 2023 23:32:41 +0200 Peter Seiderer <[email protected]> wrote:

Add purge loacle data support (inspired by BR2_ENABLE_LOCALE_PURGE handling in Makefile) to reduce target space allocation by the python-babel package, e.g. from 32MB to 24K for target/usr/lib/python3.11/site-packages/babel/locale-data with BR2_ENABLE_LOCALE_WHITELIST="C en_US de_DE".

I updated the commit log to add the explanation about en_US_POSIX, as requested by Yann.

Quoted 11 lines
---
 package/python-babel/python-babel.mk | 12 ++++++++++++
 1 file changed, 12 insertions(+)
package/python-babel/python-babel.mk +0 -0
index 890a8b02b9..ea383e9c05 100644

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 18

Quoted 7 lines
package/python-babel/python-babel.mk +4 -0
@@ -12,5 +12,17 @@ PYTHON_BABEL_LICENSE = BSD-3-Clause
 PYTHON_BABEL_LICENSE_FILES = LICENSE
 HOST_PYTHON_BABEL_DEPENDENCIES = host-python-pytz
 
+# purge locale data (if enabled), keep special en_US_POSIX data by default
+ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
+define PYTHON_BABEL_CLEANUP_LOCALE
+	for i in `ls $(TARGET_DIR)/usr/lib/python3.11/site-packages/babel/locale-data/*.dat`; \

The hardcoded 3.11 was not good, it would break at the next major bump of Python, replace with $(PYTHON3_VERSION_MAJOR).

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 25

Quoted 7 lines
package/python-babel/python-babel.mk +7 -0
+	do \
+		i_base=`basename "$$i" .dat`; \
+		echo "$(BR2_ENABLE_LOCALE_WHITELIST) en_US_POSIX" | grep -qw "$$i_base" || rm "$$i"; \
+	done
+endef
+endif
+PYTHON_BABEL_POST_INSTALL_TARGET_HOOKS += PYTHON_BABEL_CLEANUP_LOCALE

Changed to PYTHON_LABEL_TARGET_FINALIZE_HOOKS, as suggested by Yann, and moved inside the ifeq ... endif block.

Applied with those changes.

Thanks a lot!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

Hello Thomas, *,

Quoted 45 lines · Thomas Petazzoni via buildroot
Quoted 5 lines · On Tue, 18 Apr 2023 23:32:41 +0200 Peter Seiderer <[email protected]> wrote:

Add purge loacle data support (inspired by BR2_ENABLE_LOCALE_PURGE handling in Makefile) to reduce target space allocation by the python-babel package, e.g. from 32MB to 24K for target/usr/lib/python3.11/site-packages/babel/locale-data with BR2_ENABLE_LOCALE_WHITELIST="C en_US de_DE".

I updated the commit log to add the explanation about en_US_POSIX, as requested by Yann.

Quoted 11 lines
---
 package/python-babel/python-babel.mk | 12 ++++++++++++
 1 file changed, 12 insertions(+)
package/python-babel/python-babel.mk +0 -0
index 890a8b02b9..ea383e9c05 100644

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 18

Quoted 7 lines
package/python-babel/python-babel.mk +4 -0
@@ -12,5 +12,17 @@ PYTHON_BABEL_LICENSE = BSD-3-Clause
 PYTHON_BABEL_LICENSE_FILES = LICENSE
 HOST_PYTHON_BABEL_DEPENDENCIES = host-python-pytz
 
+# purge locale data (if enabled), keep special en_US_POSIX data by default
+ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
+define PYTHON_BABEL_CLEANUP_LOCALE
+	for i in `ls $(TARGET_DIR)/usr/lib/python3.11/site-packages/babel/locale-data/*.dat`; \

The hardcoded 3.11 was not good, it would break at the next major bump of Python, replace with $(PYTHON3_VERSION_MAJOR).

package/python-babel/python-babel.mk PYTHON_BABEL_LICENSE = BSD-3-Clause · line 25

Quoted 7 lines
package/python-babel/python-babel.mk +7 -0
+	do \
+		i_base=`basename "$$i" .dat`; \
+		echo "$(BR2_ENABLE_LOCALE_WHITELIST) en_US_POSIX" | grep -qw "$$i_base" || rm "$$i"; \
+	done
+endef
+endif
+PYTHON_BABEL_POST_INSTALL_TARGET_HOOKS += PYTHON_BABEL_CLEANUP_LOCALE

Changed to PYTHON_LABEL_TARGET_FINALIZE_HOOKS, as suggested by Yann, and moved inside the ifeq ... endif block.

Quoted 3 lines

Applied with those changes.

Thanks a lot!

Seems I missed to send an updated (non RFC) version, many thanks for fixing and applying....

Regards,
Peter

Quoted 2 lines

Thomas

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot