Skip to contentSkip to conversation

tests: Let xorriso fixely assume UTF-8 as local character set

PATCH In review Thomas Schmitt 3 participants 4 messages Aug 27, 2021 – Sep 8, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests: Let xorriso fixely assume UTF-8 as local character set

iso9660_test fails if the effective locale is not UTF-8. This happens because xorriso needs to convert file names and FSLABEL to UCS-2 when preparing a Joliet tree. grub-fs-tester obviously intends to use UTF-8 as character set, but xorriso assumes by default the result of nl_langinfo(3) with item CODESET.
So override the result of nl_langinfo(CODESET) by options of xorriso -as mkisofs.

---
 tests/util/grub-fs-tester.in | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
tests/util/grub-fs-tester.in +13 -9
index bfc425e1f..4f581b638 100644
@@ -8,6 +8,10 @@ GRUBFSTEST="@builddir@/grub-fstest"
 
 tempdir=`mktemp -d "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
 
+# xorriso -as mkisofs options to ignore locale when processing file names and
+# FSLABEL. This is especially needed for the conversion to Joliet UCS-2.
+XORRISOFS_CHARSET="-input-charset UTF-8 -output-charset UTF-8"
+
 # This wrapper is to ease insertion of valgrind or time statistics
 run_it () {
     LC_ALL=C "$GRUBFSTEST" "$@"
@@ -1020,31 +1024,31 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE" "$MAXLOGSECSIZE" 1); do
 		    (cd "$MASTER"; find . | cpio -o -H "$(echo ${fs} | sed 's@^cpio_@@')" > "${FSIMAGEP}0.img" ) ;;
 		x"ziso9660")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso -compliance rec_mtime -set_filter_r --zisofs -- -zisofs default -as mkisofs -iso-level 3 -graft-points -R -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img"  -- -set_filter_r --zisofs -- -zisofs default -add /="$MASTER" ;;
+		    xorriso -compliance rec_mtime -set_filter_r --zisofs -- -zisofs default -as mkisofs $XORRISOFS_CHARSET -iso-level 3 -graft-points -R -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img"  -- -set_filter_r --zisofs -- -zisofs default -add /="$MASTER" ;;
 		x"iso9660")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge off -compliance rec_mtime -as mkisofs -iso-level 3 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge off -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 3 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"joliet")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge off  -compliance rec_mtime -as mkisofs -iso-level 3 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge off  -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 3 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"rockridge")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge on -compliance rec_mtime -as mkisofs -iso-level 3 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge on -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 3 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"rockridge_joliet")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge on -compliance rec_mtime -as mkisofs -iso-level 3 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge on -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 3 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"iso9660_1999")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge off -compliance rec_mtime -as mkisofs -iso-level 4 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge off -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 4 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"joliet_1999")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge off  -compliance rec_mtime -as mkisofs -iso-level 4 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge off  -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 4 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"rockridge_1999")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge on -compliance rec_mtime -as mkisofs -iso-level 4 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge on -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 4 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"rockridge_joliet_1999")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso --rockridge on -compliance rec_mtime -as mkisofs -iso-level 4 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
+		    xorriso --rockridge on -compliance rec_mtime -as mkisofs $XORRISOFS_CHARSET -iso-level 4 -graft-points -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;
 		x"romfs")
 		    genromfs -V "$FSLABEL" -f "${FSIMAGEP}0.img" -d "$MASTER" ;;
 		xsquash4_*)

--
2.20.1

Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Thomas Schmitt

iso9660_test fails if the effective locale is not UTF-8. This happens because xorriso needs to convert file names and FSLABEL to UCS-2 when preparing a Joliet tree. grub-fs-tester obviously intends to use UTF-8 as character set, but xorriso assumes by default the result of nl_langinfo(3) with item CODESET.
So override the result of nl_langinfo(CODESET) by options of xorriso -as mkisofs.

Thomas, Glenn, thank you for doing such deep investigation!

Glenn, did you test this patch?

Daniel

#2 lore raw
Quoted 9 lines · On Wed, 1 Sep 2021 15:11:38 +0200 Daniel Kiper <[email protected]> wrote:
Quoted 8 lines · Thomas Schmitt

iso9660_test fails if the effective locale is not UTF-8. This happens because xorriso needs to convert file names and FSLABEL to UCS-2 when preparing a Joliet tree. grub-fs-tester obviously intends to use UTF-8 as character set, but xorriso assumes by default the result of nl_langinfo(3) with item CODESET. So override the result of nl_langinfo(CODESET) by options of xorriso -as mkisofs.

Quoted 7 lines
Quoted 1 line

Thomas, Glenn, thank you for doing such deep investigation!

Glenn, did you test this patch?

Coming to this late, had some issues which kept me away. I notice that the patch is already in master, but for what its worth, I have now tested with the patch and LANG unset and the test now succeeds. Thanks Thomas.

Glenn

#3 lore raw
Quoted 14 lines · Glenn Washburn
Quoted 11 lines · On Wed, 1 Sep 2021 15:11:38 +0200 Daniel Kiper <[email protected]> wrote:
Quoted 9 lines · Thomas Schmitt

iso9660_test fails if the effective locale is not UTF-8. This happens because xorriso needs to convert file names and FSLABEL to UCS-2 when preparing a Joliet tree. grub-fs-tester obviously intends to use UTF-8 as character set, but xorriso assumes by default the result of nl_langinfo(3) with item CODESET. So override the result of nl_langinfo(CODESET) by options of xorriso -as mkisofs.

Quoted 7 lines
Quoted 5 lines

Thomas, Glenn, thank you for doing such deep investigation!

Glenn, did you test this patch?

Coming to this late, had some issues which kept me away. I notice that

No worries...

Quoted 2 lines

the patch is already in master, but for what its worth, I have now tested with the patch and LANG unset and the test now succeeds. Thanks

Thank you for doing test.

Daniel