Skip to contentSkip to conversation

icu: Add knobs to generate a subset of ICU data

PATCH Superseded Khem Raj 2 participants 5 messages Apr 3, 2020

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

PATCH 1/1 icu: Add knobs to generate a subset of ICU data

Recent versions of ICU (64+) provides a tool for configuring ICU locale data file with finer granularity [1]

Default generated size for libicudata.so.66.1 is ~27M, which is quite large for embedded systems and all of them may not even need all locale data.

This patch calls the icudata buildtool during configure on the icudata and utilizes a filter called 'filter.json` ( empty by default) to create the data, default behavior should remain same but someone can add a filter.json in own layer to configure this data, e.g.

{

  "localeFilter": {
    "filterType": "language",
    "whitelist": [
      "en",
      "de",
      "it"
    ]

}
}

would only generate the locale data for english/german/italian

This would reduce the size of libicudata.so.66.1 to 12M

[1] https://github.com/unicode-org/icu/blob/master/docs/userguide/icu_data/buildtool.md

---
 meta/recipes-support/icu/icu/filter.json |  2 ++
 meta/recipes-support/icu/icu_66.1.bb     | 19 ++++++++++++++++---
 2 files changed, 18 insertions(+), 3 deletions(-)
 create mode 100644 meta/recipes-support/icu/icu/filter.json
meta/recipes-support/icu/icu/filter.json new file +2 -0
new file mode 100644
index 0000000000..2c63c08510
@@ -0,0 +1,2 @@
+{
+}
meta/recipes-support/icu/icu_66.1.bb +16 -3
index 5018464c14..a8cfdf9eed 100644
@@ -18,7 +18,10 @@ ARM_INSTRUCTION_SET_armv4 = "arm"
 ARM_INSTRUCTION_SET_armv5 = "arm"
 
 BASE_SRC_URI = "https://github.com/unicode-org/icu/releases/download/release-${ICU_FOLDER}/icu4c-${ICU_PV}-src.tgz"
-SRC_URI = "${BASE_SRC_URI} \
+DATA_SRC_URI = "https://github.com/unicode-org/icu/releases/download/release-${ICU_FOLDER}/icu4c-${ICU_PV}-data.zip"
+SRC_URI = "${BASE_SRC_URI};name=code \
+           ${DATA_SRC_URI};name=data \
+           file://filter.json \
            file://icu-pkgdata-large-cmd.patch \
            file://fix-install-manx.patch \
            file://0001-Fix-big-endian-build.patch \
@@ -28,8 +31,18 @@ SRC_URI = "${BASE_SRC_URI} \
 SRC_URI_append_class-target = "\
            file://0001-Disable-LDFLAGSICUDT-for-Linux.patch \
           "
-SRC_URI[md5sum] = "b33dc6766711517c98d318447e5110f8"
-SRC_URI[sha256sum] = "52a3f2209ab95559c1cf0a14f24338001f389615bf00e2585ef3dbc43ecf0a2e"
+SRC_URI[code.sha256sum] = "52a3f2209ab95559c1cf0a14f24338001f389615bf00e2585ef3dbc43ecf0a2e"
+SRC_URI[data.sha256sum] = "8be647f738891d2beb79d48f99077b3499948430eae6f1be112553b15ab0243e"
 
 UPSTREAM_CHECK_REGEX = "icu4c-(?P<pver>\d+(_\d+)+)-src"
 UPSTREAM_CHECK_URI = "https://github.com/unicode-org/icu/releases"
+
+CONFIGURE_SCRIPT_class-target = "${S}/runConfigureICU"
+CONFIGUREOPTS_prepend_class-target = "Linux "
+CACHED_CONFIGUREVARS_append_class-target = "ICU_DATA_FILTER_FILE=${WORKDIR}/filter.json"
+
+do_configure_prepend_class-target () {
+    rm -rf ${S}/data
+    cp -a ${WORKDIR}/data ${S}/
+}
+
Signature · 1 line
2.26.0
#1 lore raw
Quoted 8 lines · Khem Raj

Recent versions of ICU (64+) provides a tool for configuring ICU locale data file with finer granularity [1]

Default generated size for libicudata.so.66.1 is ~27M, which is quite large for embedded systems and all of them may not even need all locale
data.

Quoted 5 lines

This patch calls the icudata buildtool during configure on the icudata
and utilizes a filter called 'filter.json` ( empty by default) to create the data, default behavior should remain same but someone can add a filter.json in own layer to configure this data, e.g.

Nice idea, fails on ppc:

https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/411

Cheers,

Richard

author
#2 lore raw
Quoted 16 lines · Richard Purdie
Quoted 8 lines · Khem Raj

Recent versions of ICU (64+) provides a tool for configuring ICU locale data file with finer granularity [1]

Default generated size for libicudata.so.66.1 is ~27M, which is quite large for embedded systems and all of them may not even need all locale
data.

Quoted 5 lines

This patch calls the icudata buildtool during configure on the icudata
and utilizes a filter called 'filter.json` ( empty by default) to create the data, default behavior should remain same but someone can add a filter.json in own layer to configure this data, e.g.

Quoted 4 lines

Nice idea, fails on ppc:

https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/411

yeah looking into it. Are there any other arch failures ?

Quoted 4 lines

Cheers,

Richard

#3 lore raw
Quoted 19 lines · Khem Raj
Quoted 17 lines · Richard Purdie
Quoted 15 lines · Khem Raj

Recent versions of ICU (64+) provides a tool for configuring ICU locale data file with finer granularity [1]

Default generated size for libicudata.so.66.1 is ~27M, which is quite
large for embedded systems and all of them may not even need all locale
data.

This patch calls the icudata buildtool during configure on the icudata
and utilizes a filter called 'filter.json` ( empty by default) to create the data, default behavior should remain same but someone can
add a filter.json in own layer to configure this data, e.g.

Quoted 6 lines
Quoted 4 lines

Nice idea, fails on ppc:

https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/411

yeah looking into it. Are there any other arch failures ?

mips:
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/1749
mips64:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/1753
edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/414

Cheers,

Richard

author
#4 lore raw
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Khem Raj
Quoted 21 lines · Richard Purdie
Quoted 9 lines · Khem Raj

Recent versions of ICU (64+) provides a tool for configuring ICU locale data file with finer granularity [1]

Default generated size for libicudata.so.66.1 is ~27M, which is quite
large for embedded systems and all of them may not even need all locale
data.

Quoted 6 lines

This patch calls the icudata buildtool during configure on the icudata
and utilizes a filter called 'filter.json` ( empty by default) to create the data, default behavior should remain same but someone can
add a filter.json in own layer to configure this data, e.g.

Nice idea, fails on ppc:

https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/411

Quoted 9 lines
Quoted 1 line

yeah looking into it. Are there any other arch failures ?

mips:
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/1749
mips64:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/1753
edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/414

Thanks, I think I understand the problem, its when target endianness is bigendian
and the host is little-endian, I have sent a v2 which should fix it. Give v2 a shot.

Quoted 5 lines

Cheers,

Richard