fs: Remove usage of broken nls_utf8 and drop it
Subsystems (10)
v2 cover fs: Remove usage of broken nls_utf8 and drop it
Module nls_utf8 is broken in several ways. It does not support (full) UTF-8, despite its name. It cannot handle 4-byte UTF-8 sequences and tolower/toupper table is not implemented at all. Which means that it is not suitable for usage in case-insensitive filesystems or UTF-16 filesystems (because of e.g. missing UTF-16 surrogate pairs processing).
This is RFC v2 patch series which unify and fix iocharset=utf8 mount option in all fs drivers and converts all remaining fs drivers to use utf8s_to_utf16s(), utf16s_to_utf8s(), utf8_to_utf32(), utf32_to_utf8 functions for implementing UTF-8 support instead of nls_utf8.
So at the end it allows to completely drop this broken nls_utf8 module.
For more details look at email thread where was discussed fs unification:
https://lore.kernel.org/linux-fsdevel/20200102211855.gg62r7jshp742d6i@pali/t/#u
This patch series is mostly untested and presented as RFC. Please let me know what do you think about it and if is the correct way how to fix broken UTF-8 support in fs drivers. As explained in above email thread I think it does not make sense to try fixing whole NLS framework and it is easier to just drop this nls_utf8 module.
Note: this patch series does not address UTF-8 fat case-sensitivity issue:
https://lore.kernel.org/linux-fsdevel/20200119221455.bac7dc55g56q2l4r@pali/
Changes since RFC v1:
* Dropped already merged udf and isofs patches
* Addressed review comments:
- updated documentation
- usage of seq_puts
- some code moved to local variables
- usage of true/false instead of 1/0
- rebased on top of master branch
Link to RFC v1:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Pali Rohár (18):
fat: Fix iocharset=utf8 mount option
hfsplus: Add iocharset= mount option as alias for nls=
ntfs: Undeprecate iocharset= mount option
ntfs: Fix error processing when load_nls() fails
befs: Fix printing iocharset= mount option
befs: Rename enum value Opt_charset to Opt_iocharset to match mount
option
befs: Fix error processing when load_nls() fails
befs: Allow to use native UTF-8 mode
hfs: Explicitly set hsb->nls_disk when hsb->nls_io is set
hfs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
hfsplus: Do not use broken utf8 NLS table for iocharset=utf8 mount
option
jfs: Remove custom iso8859-1 implementation
jfs: Fix buffer overflow in jfs_strfromUCS_le() function
jfs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
ntfs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
cifs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
cifs: Remove usage of load_nls_default() calls
nls: Drop broken nls_utf8 module
Documentation/filesystems/hfsplus.rst | 3 + Documentation/filesystems/ntfs.rst | 5 +- Documentation/filesystems/vfat.rst | 13 +-- fs/befs/linuxvfs.c | 24 +++-- fs/cifs/cifs_unicode.c | 128 +++++++++++++++++--------- fs/cifs/cifs_unicode.h | 2 +- fs/cifs/cifsfs.c | 2 + fs/cifs/cifssmb.c | 8 +- fs/cifs/connect.c | 8 +- fs/cifs/dfs_cache.c | 24 ++--- fs/cifs/dir.c | 28 ++++-- fs/cifs/smb2pdu.c | 18 +--- fs/cifs/winucase.c | 14 ++- fs/fat/Kconfig | 19 +--- fs/fat/dir.c | 17 ++-- fs/fat/fat.h | 22 +++++ fs/fat/inode.c | 28 +++--- fs/fat/namei_vfat.c | 26 ++++-- fs/hfs/super.c | 62 +++++++++++-- fs/hfs/trans.c | 62 +++++++------ fs/hfsplus/dir.c | 7 +- fs/hfsplus/options.c | 39 +++++--- fs/hfsplus/super.c | 7 +- fs/hfsplus/unicode.c | 31 ++++++- fs/hfsplus/xattr.c | 20 ++-- fs/hfsplus/xattr_security.c | 6 +- fs/jfs/jfs_dtree.c | 13 ++- fs/jfs/jfs_unicode.c | 35 +++---- fs/jfs/jfs_unicode.h | 2 +- fs/jfs/super.c | 29 ++++-- fs/nls/Kconfig | 9 -- fs/nls/Makefile | 1 - fs/nls/nls_utf8.c | 67 -------------- fs/ntfs/dir.c | 6 +- fs/ntfs/inode.c | 5 +- fs/ntfs/super.c | 60 ++++++------ fs/ntfs/unistr.c | 29 +++++- 37 files changed, 493 insertions(+), 386 deletions(-) delete mode 100644 fs/nls/nls_utf8.c
Signature · 1 line
2.20.1
RFC PATCH v2 1/18 fat: Fix iocharset=utf8 mount option
Currently iocharset=utf8 mount option is broken and error is printed to dmesg when it is used. To use UTF-8 as iocharset, it is required to use utf8=1 mount option.
Fix iocharset=utf8 mount option to use be equivalent to the utf8=1 mount option and remove printing error from dmesg.
FAT by definition is case-insensitive but current Linux implementation is case-sensitive for non-ASCII characters when UTF-8 is used. This patch does not change this UTF-8 behavior. Only more comments in fat_utf8_strnicmp() function are added about it.
After this patch iocharset=utf8 starts working, so there is no need to have separate config option FAT_DEFAULT_UTF8 as FAT_DEFAULT_IOCHARSET for utf8 also starts working. So remove redundant config option FAT_DEFAULT_UTF8.
--- Documentation/filesystems/vfat.rst | 13 ++----------- fs/fat/Kconfig | 19 +------------------ fs/fat/dir.c | 17 +++++++---------- fs/fat/fat.h | 22 ++++++++++++++++++++++ fs/fat/inode.c | 28 +++++++++++----------------- fs/fat/namei_vfat.c | 26 +++++++++++++++++++------- 6 files changed, 62 insertions(+), 63 deletions(-)
Documentation/filesystems/vfat.rst +2 -11
@@ -65,19 +65,10 @@ VFAT MOUNT OPTIONS in Unicode format, but Unix for the most part doesn't know how to deal with Unicode. By default, FAT_DEFAULT_IOCHARSET setting is used. - - There is also an option of doing UTF-8 translations - with the utf8 option. - -.. note:: ``iocharset=utf8`` is not recommended. If unsure, you should consider - the utf8 option instead. + **utf8** is supported too and recommended to use. **utf8=<bool>** - UTF-8 is the filesystem safe version of Unicode that - is used by the console. It can be enabled or disabled - for the filesystem with this option. - If 'uni_xlate' gets set, UTF-8 gets disabled. - By default, FAT_DEFAULT_UTF8 setting is used. + Alias for ``iocharset=utf8`` mount option. **uni_xlate=<bool>** Translate unhandled Unicode characters to special
fs/fat/Kconfig +1 -18
@@ -93,29 +93,12 @@ config FAT_DEFAULT_IOCHARSET like FAT to use. It should probably match the character set that most of your FAT filesystems use, and can be overridden with the "iocharset" mount option for FAT filesystems. - Note that "utf8" is not recommended for FAT filesystems. - If unsure, you shouldn't set "utf8" here - select the next option - instead if you would like to use UTF-8 encoded file names by default. + "utf8" is supported too and recommended to use. See <file:Documentation/filesystems/vfat.rst> for more information. Enable any character sets you need in File Systems/Native Language Support. -config FAT_DEFAULT_UTF8 - bool "Enable FAT UTF-8 option by default" - depends on VFAT_FS - default n - help - Set this if you would like to have "utf8" mount option set - by default when mounting FAT filesystems. - - Even if you say Y here can always disable UTF-8 for - particular mount by adding "utf8=0" to mount options. - - Say Y if you use UTF-8 encoding for file names, N otherwise. - - See <file:Documentation/filesystems/vfat.rst> for more information. - config FAT_KUNIT_TEST tristate "Unit Tests for FAT filesystems" if !KUNIT_ALL_TESTS depends on KUNIT && FAT_FS
fs/fat/dir.c +7 -10
@@ -33,11 +33,6 @@#define FAT_MAX_UNI_CHARS ((MSDOS_SLOTS - 1) * 13 + 1)#define FAT_MAX_UNI_SIZE (FAT_MAX_UNI_CHARS * sizeof(wchar_t)) -static inline unsigned char fat_tolower (unsigned char c) -{ -return ((c >='A' ) && (c <='Z' )) ? c+32 : c; -} -static inline loff_t fat_make_i_pos (struct super_block *sb,struct buffer_head *bh,struct msdos_dir_entry *de)
@@ -258,10 +253,12 @@ static inline int fat_name_match(struct msdos_sb_info *sbi,if (a_len != b_len)return 0 ; -if (sbi->options.name_check !='s' ) -return !nls_strnicmp (sbi->nls_io, a, b, a_len); -else +if (sbi->options.name_check =='s' )return !memcmp (a, b, a_len); +else if (sbi->options.utf8) +return !fat_utf8_strnicmp (a, b, a_len); +else +return !nls_strnicmp (sbi->nls_io, a, b, a_len); }enum {PARSE_INVALID =1 ,PARSE_NOT_LONGNAME ,PARSE_EOF , };
@@ -384,7 +381,7 @@ static int fat_parse_short(struct super_block *sb, de->lcase & CASE_LOWER_BASE);if (chl <=1 ) {if (!isvfat) - ptname[i] = nocase ? c :fat_tolower (c); + ptname[i] = nocase ? c :fat_ascii_to_lower (c); i++;if (c !=' ' ) { name_len = i;
@@ -421,7 +418,7 @@ static int fat_parse_short(struct super_block *sb,if (chl <=1 ) { k++;if (!isvfat) - ptname[i] = nocase ? c :fat_tolower (c); + ptname[i] = nocase ? c :fat_ascii_to_lower (c); i++;if (c !=' ' ) { name_len = i;
fs/fat/fat.h +22 -0
@@ -311,6 +311,28 @@ static inline void fatwchar_to16(__u8 *dst, const wchar_t *src, size_t len)#endif } +static inline unsigned char fat_ascii_to_lower (unsigned char c) +{ +return ((c >='A' ) && (c <='Z' )) ? c+32 : c; +} + +static inline int fat_utf8_strnicmp (const unsigned char *a, +const unsigned char *b, +int len) +{ +int i; + +/* +* FIXME: UTF-8 doesn't provide FAT semantics +* Case-insensitive support is only for 7-bit ASCII characters +*/ +for (i =0 ; i < len; i++) { +if (fat_ascii_to_lower (a[i]) !=fat_ascii_to_lower (b[i])) +return 1 ; + } +return 0 ; +} +/* fat/cache.c */ extern void fat_cache_inval_inode (struct inode *inode); extern int fat_get_cluster(struct inode *inode, int cluster,
fs/fat/inode.c +11 -17
@@ -956,7 +956,9 @@ static int fat_show_options(struct seq_file *m, struct dentry *root)/* strip "cp" prefix from displayed option */ seq_printf (m,",codepage=%s" , &sbi->nls_disk->charset[2 ]);if (isvfat) { -if (sbi->nls_io) +if (opts->utf8) +seq_puts (m,",iocharset=utf8" ); +else if (sbi->nls_io)seq_printf (m,",iocharset=%s" , sbi->nls_io->charset);switch (opts->shortname) {
@@ -993,8 +995,6 @@ static int fat_show_options(struct seq_file *m, struct dentry *root)if (opts->nocase)seq_puts (m,",nocase" ); }else { -if (opts->utf8) -seq_puts (m,",utf8" );if (opts->unicode_xlate)seq_puts (m,",uni_xlate" );if (!opts->numtail)
@@ -1156,8 +1156,6 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, opts->errors =FAT_ERRORS_RO ; *debug =0 ; - opts->utf8 =IS_ENABLED (CONFIG_FAT_DEFAULT_UTF8 ) && is_vfat; -if (!options) goto out;
@@ -1318,10 +1316,14 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, | VFAT_SFN_CREATE_WIN95;break ;case Opt_utf8_no:/* 0 or no or false */ - opts->utf8 =0 ; +fat_reset_iocharset (opts);break ;case Opt_utf8_yes:/* empty or 1 or yes or true */ - opts->utf8 =1 ; +fat_reset_iocharset (opts); + iocharset =kstrdup ("utf8" ,GFP_KERNEL ); +if (!iocharset) +return -ENOMEM ; + opts->iocharset = iocharset;break ;case Opt_uni_xl_no:/* 0 or no or false */ opts->unicode_xlate =0 ;
@@ -1359,18 +1361,11 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, } out: -/* UTF-8 doesn't provide FAT semantics */ -if (!strcmp (opts->iocharset,"utf8" )) { -fat_msg (sb,KERN_WARNING ,"utf8 is not a recommended IO charset" -" for FAT filesystems, filesystem will be " -"case sensitive!" ); - } + opts->utf8 = !strcmp (opts->iocharset,"utf8" ) && is_vfat;/* If user doesn't specify allow_utime, it's initialized from dmask. */ if (opts->allow_utime == (unsigned short )-1 ) opts->allow_utime = ~opts->fs_dmask & (S_IWGRP |S_IWOTH ); -if (opts->unicode_xlate) - opts->utf8 =0 ;if (opts->nfs ==FAT_NFS_NOSTALE_RO ) { sb->s_flags |=SB_RDONLY ; sb->s_export_op = &fat_export_ops_nostale;
@@ -1828,8 +1823,7 @@ int fat_fill_super(struct super_block *sb, void *data, int silent, int isvfat, goto out_fail; } -/* FIXME: utf8 is using iocharset for upper/lower conversion */ -if (sbi->options.isvfat) { +if (sbi->options.isvfat && !sbi->options.utf8) { sbi->nls_io =load_nls (sbi->options.iocharset);if (!sbi->nls_io) { fat_msg(sb, KERN_ERR, "IO charset %s not found",
fs/fat/namei_vfat.c +19 -7
@@ -134,6 +134,7 @@ static int vfat_hash(const struct dentry *dentry, struct qstr *qstr)static int vfat_hashi (const struct dentry *dentry,struct qstr *qstr) {struct nls_table *t =MSDOS_SB (dentry->d_sb)->nls_io; +int utf8 =MSDOS_SB (dentry->d_sb)->options.utf8;const unsigned char *name;unsigned int len;unsigned long hash;
@@ -142,8 +143,17 @@ static int vfat_hashi(const struct dentry *dentry, struct qstr *qstr) len =vfat_striptail_len (qstr); hash =init_name_hash (dentry); -while (len--) - hash =partial_name_hash (nls_tolower (t, *name++), hash); +if (utf8) { +/* +* FIXME: UTF-8 doesn't provide FAT semantics +* Case-insensitive support is only for 7-bit ASCII characters +*/ +while (len--) + hash =partial_name_hash (fat_ascii_to_lower (*name++), hash); + }else { +while (len--) + hash =partial_name_hash (nls_tolower (t, *name++), hash); + } qstr->hash =end_name_hash (hash);return 0 ;
@@ -156,16 +166,18 @@ static int vfat_cmpi(const struct dentry *dentry, unsigned int len, const char *str, const struct qstr *name) { struct nls_table *t = MSDOS_SB(dentry->d_sb)->nls_io; +int utf8 =MSDOS_SB (dentry->d_sb)->options.utf8;unsigned int alen, blen;/* A filename cannot end in '.' or we treat it like it has none */ alen =vfat_striptail_len (name); blen =__vfat_striptail_len (len, str); -if (alen == blen) { -if (nls_strnicmp (t, name->name, str, alen) ==0 ) -return 0 ; - } -return 1 ; +if (alen != blen) +return 1 ; +else if (utf8) +return fat_utf8_strnicmp (name->name, str, alen); +else +return nls_strnicmp (t, name->name, str, alen); }/*
Signature · 1 line
2.20.1
RFC PATCH v2 2/18 hfsplus: Add iocharset= mount option as alias for nls=
In reply to Pali RohárOther fs drivers are using iocharset= mount option for specifying charset. So add it also for hfsplus and mark old nls= mount option as deprecated.
--- Documentation/filesystems/hfsplus.rst | 3 +++ fs/hfsplus/options.c | 7 ++++++- 2 files changed, 9 insertions(+), 1 deletion(-)
Documentation/filesystems/hfsplus.rst +3 -0
@@ -50,6 +50,9 @@ When mounting an HFSPlus filesystem, the following options are accepted: or locked. Use at your own risk. nls=cccc + Deprecated alias for ``iocharset=`` mount option. + + iocharset=cccc Encoding to use when presenting file names.
fs/hfsplus/options.c +6 -1
@@ -23,6 +23,7 @@ enum { opt_creator, opt_type, opt_umask, opt_uid, opt_gid, opt_part, opt_session, opt_nls, + opt_iocharset, opt_nodecompose, opt_decompose, opt_barrier, opt_nobarrier, opt_force, opt_err
@@ -37,6 +38,7 @@ static const match_table_t tokens = { { opt_part, "part=%u" }, { opt_session, "session=%u" }, { opt_nls, "nls=%s" }, + { opt_iocharset, "iocharset=%s" }, { opt_decompose, "decompose" }, { opt_nodecompose, "nodecompose" }, { opt_barrier, "barrier" },
@@ -170,6 +172,9 @@ int hfsplus_parse_options(char *input, struct hfsplus_sb_info *sbi) }break ;case opt_nls: +pr_warn ("option nls= is deprecated, use iocharset=\n" ); + fallthrough; +case opt_iocharset:if (sbi->nls) {pr_err ("unable to change nls mapping\n" );return 0 ;
@@ -234,7 +239,7 @@ int hfsplus_show_options(struct seq_file *seq, struct dentry *root)if (sbi->session >=0 )seq_printf (seq,",session=%u" , sbi->session);if (sbi->nls) -seq_printf (seq,",nls=%s" , sbi->nls->charset); +seq_printf (seq,",iocharset=%s" , sbi->nls->charset);if (test_bit (HFSPLUS_SB_NODECOMPOSE , &sbi->flags))seq_puts (seq,",nodecompose" );if (test_bit (HFSPLUS_SB_NOBARRIER , &sbi->flags))
Signature · 1 line
2.20.1
RFC PATCH v2 4/18 ntfs: Fix error processing when load_nls() fails
In reply to Pali RohárEnsure that specified charset in iocharset= mount option is used. On error correctly propagate error code back to the caller.
--- fs/ntfs/super.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-)
fs/ntfs/super.c +5 -13
@@ -94,7 +94,7 @@ static bool parse_options(ntfs_volume *vol, char *opt)umode_t fmask = (umode_t)-1 , dmask = (umode_t)-1 ;int mft_zone_multiplier =-1 , on_errors =-1 ;int show_sys_files =-1 , case_sensitive =-1 , disable_sparse =-1 ; -struct nls_table *nls_map =NULL , *old_nls; +struct nls_table *nls_map =NULL ;/* I am lazy... (-8 */ #define NTFS_GETOPT_WITH_DEFAULT (option, variable, default_value) \
@@ -195,20 +195,12 @@ static bool parse_options(ntfs_volume *vol, char *opt)if (!v || !*v) goto needs_arg; use_utf8: - old_nls = nls_map; +unload_nls (nls_map); nls_map =load_nls (v);if (!nls_map) { -if (!old_nls) { -ntfs_error (vol->sb,"NLS character set " -"%s not found." , v); -return false; - } -ntfs_error (vol->sb,"NLS character set %s not " -"found. Using previous one %s." , - v, old_nls->charset); - nls_map = old_nls; - }else /* nls_map */ { -unload_nls (old_nls); +ntfs_error (vol->sb,"NLS character set " +"%s not found." , v); +return false; } }else if (!strcmp (p,"utf8" )) {bool val = false;
Signature · 1 line
2.20.1
RFC PATCH v2 6/18 befs: Rename enum value Opt_charset to Opt_iocharset to match mount option
In reply to Pali RohárMount option is named iocharset= and not charset=
--- fs/befs/linuxvfs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/befs/linuxvfs.c +3 -3
@@ -677,13 +677,13 @@ static struct dentry *befs_get_parent(struct dentry *child) }enum { - Opt_uid, Opt_gid, Opt_charset, Opt_debug, Opt_err, + Opt_uid, Opt_gid, Opt_iocharset, Opt_debug, Opt_err, };static const match_table_t befs_tokens = { {Opt_uid,"uid=%d" }, {Opt_gid,"gid=%d" }, - {Opt_charset,"iocharset=%s" }, + {Opt_iocharset,"iocharset=%s" }, {Opt_debug,"debug" }, {Opt_err,NULL } };
@@ -744,7 +744,7 @@ parse_options(char *options, struct befs_mount_options *opts) opts->gid = gid; opts->use_gid =1 ;break ; -case Opt_charset: +case Opt_iocharset:kfree (opts->iocharset); opts->iocharset =match_strdup (&args[0 ]);if (!opts->iocharset) {
Signature · 1 line
2.20.1
RFC PATCH v2 3/18 ntfs: Undeprecate iocharset= mount option
In reply to Pali RohárOther fs drivers are using iocharset= mount option for specifying charset. So mark iocharset= mount option as preferred and deprecate nls= mount option.
--- Documentation/filesystems/ntfs.rst | 5 ++--- fs/ntfs/inode.c | 2 +- fs/ntfs/super.c | 13 ++++--------- fs/ntfs/unistr.c | 3 ++- 4 files changed, 9 insertions(+), 14 deletions(-)
Documentation/filesystems/ntfs.rst +2 -3
@@ -109,10 +109,9 @@ mount command (man 8 mount, also see man 5 fstab), the NTFS driver supports the following mount options: ======================= ======================================================= -iocharset=name Deprecated option. Still supported but please use - nls=name in the future. See description for nls=name. +nls=name Alias for ``iocharset=`` mount option. -nls=name Character set to use when returning file names. +iocharset=name Character set to use when returning file names. Unlike VFAT, NTFS suppresses names that contain unconvertible characters. Note that most character sets contain insufficient characters to represent all
fs/ntfs/inode.c +1 -1
@@ -2314,7 +2314,7 @@ int ntfs_show_options(struct seq_file *sf, struct dentry *root)seq_printf (sf,",fmask=0%o" , vol->fmask);seq_printf (sf,",dmask=0%o" , vol->dmask); } -seq_printf (sf,",nls=%s" , vol->nls_map->charset); +seq_printf (sf,",iocharset=%s" , vol->nls_map->charset);if (NVolCaseSensitive (vol))seq_printf (sf,",case_sensitive" );if (NVolShowSystemFiles (vol))
fs/ntfs/super.c +4 -9
@@ -192,11 +192,6 @@ static bool parse_options(ntfs_volume *vol, char *opt)ntfs_warning (vol->sb,"Ignoring obsolete option %s." , p); else if (!strcmp(p, "nls") || !strcmp(p, "iocharset")) { -if (!strcmp (p,"iocharset" )) -ntfs_warning (vol->sb,"Option iocharset is " -"deprecated. Please use " -"option nls=<charsetname> in " -"the future." );if (!v || !*v) goto needs_arg; use_utf8:
@@ -218,10 +213,10 @@ static bool parse_options(ntfs_volume *vol, char *opt) }else if (!strcmp (p,"utf8" )) {bool val = false;ntfs_warning (vol->sb,"Option utf8 is no longer " -"supported, using option nls=utf8. Please " -"use option nls=utf8 in the future and " -"make sure utf8 is compiled either as a " -"module or into the kernel." ); +"supported, using option iocharset=utf8. " +"Please use option iocharset=utf8 in the " +"future and make sure utf8 is compiled " +"either as a module or into the kernel." );if (!v || !*v) val = true;else if (!simple_getbool (v, &val))
fs/ntfs/unistr.c +2 -1
@@ -372,7 +372,8 @@ retry: wc = nls->uni2char(le16_to_cpu(ins[i]), ns + o, conversion_err:ntfs_error (vol->sb,"Unicode name contains characters that cannot be " "converted to character set %s. You might want to " -"try to use the mount option nls=utf8." , nls->charset); +"try to use the mount option iocharset=utf8." , + nls->charset);if (ns != *outs)kfree (ns);if (wc != -ENAMETOOLONG )
Signature · 1 line
2.20.1
RFC PATCH v2 5/18 befs: Fix printing iocharset= mount option
In reply to Pali RohárMount option is named iocharset= and not charset=
--- fs/befs/linuxvfs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/befs/linuxvfs.c +1 -1
@@ -777,7 +777,7 @@ static int befs_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",gid=%u" ,from_kgid_munged (&init_user_ns, opts->gid));if (opts->iocharset) -seq_printf (m,",charset=%s" , opts->iocharset); +seq_printf (m,",iocharset=%s" , opts->iocharset);if (opts->debug)seq_puts (m,",debug" );return 0 ;
Signature · 1 line
2.20.1
RFC PATCH v2 7/18 befs: Fix error processing when load_nls() fails
In reply to Pali RohárEnsure that specified charset in iocharset= mount option is used. On error correctly propagate error code back to the caller.
--- fs/befs/linuxvfs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
fs/befs/linuxvfs.c +2 -3
@@ -913,10 +913,9 @@ befs_fill_super(struct super_block *sb, void *data, int silent) befs_sb->mount_opts.iocharset); befs_sb->nls =load_nls (befs_sb->mount_opts.iocharset);if (!befs_sb->nls) { -befs_warning (sb,"Cannot load nls %s" -" loading default nls" , +befs_error (sb,"Cannot load nls %s" , befs_sb->mount_opts.iocharset); - befs_sb->nls =load_nls_default (); + goto unacquire_priv_sbp; }/* load default nls if none is specified in mount options */ }else {
Signature · 1 line
2.20.1
RFC PATCH v2 8/18 befs: Allow to use native UTF-8 mode
In reply to Pali Rohárbefs driver already has a code which avoids usage of NLS when befs_sb->nls is not set.
But befs_fill_super() always set befs_sb->nls, so activating native UTF-8 is not possible.
Fix it by not setting befs_sb->nls when iocharset is set to utf8. So now after this cgange mount option iocharset=utf8 activates usage of native UTF-8 code path in befs driver.
--- fs/befs/linuxvfs.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
fs/befs/linuxvfs.c +10 -3
@@ -769,6 +769,7 @@ static int befs_show_options(struct seq_file *m, struct dentry *root) {struct befs_sb_info *befs_sb =BEFS_SB (root->d_sb);struct befs_mount_options *opts = &befs_sb->mount_opts; +struct nls_table *nls = befs_sb->nls;if (!uid_eq (opts->uid,GLOBAL_ROOT_UID )) seq_printf(m, ",uid=%u",
@@ -776,8 +777,10 @@ static int befs_show_options(struct seq_file *m, struct dentry *root)if (!gid_eq (opts->gid,GLOBAL_ROOT_GID ))seq_printf (m,",gid=%u" ,from_kgid_munged (&init_user_ns, opts->gid)); -if (opts->iocharset) -seq_printf (m,",iocharset=%s" , opts->iocharset); +if (nls) +seq_printf (m,",iocharset=%s" , nls->charset); +else +seq_puts (m,",iocharset=utf8" );if (opts->debug)seq_puts (m,",debug" );return 0 ;
@@ -815,6 +818,7 @@ befs_fill_super(struct super_block *sb, void *data, int silent)const unsigned long sb_block =0 ;const off_t x86_sb_off =512 ;int blocksize; +const char *p; sb->s_fs_info =kzalloc (sizeof (*befs_sb),GFP_KERNEL );if (sb->s_fs_info ==NULL )
@@ -907,8 +911,11 @@ befs_fill_super(struct super_block *sb, void *data, int silent) goto unacquire_priv_sbp; } + p = befs_sb->mount_opts.iocharset ? befs_sb->mount_opts.iocharset :CONFIG_NLS_DEFAULT ; +if (strcmp (p,"utf8" ) ==0 ) { +befs_debug (sb,"Using native UTF-8 without nls" );/* load nls library */ -if (befs_sb->mount_opts.iocharset) { + }else if (befs_sb->mount_opts.iocharset) {befs_debug (sb,"Loading nls: %s" , befs_sb->mount_opts.iocharset); befs_sb->nls =load_nls (befs_sb->mount_opts.iocharset);
Signature · 1 line
2.20.1
RFC PATCH v2 9/18 hfs: Explicitly set hsb->nls_disk when hsb->nls_io is set
In reply to Pali RohárIt does not make any sense to set hsb->nls_io (NLS iocharset used between VFS and hfs driver) when hsb->nls_disk (NLS codepage used between hfs driver and disk) is not set.
Reverse engineering driver code shown what is doing in this special case:
When codepage was not defined but iocharset was then hfs driver copied 8bit character from disk directly to 16bit unicode wchar_t type. Which means it did conversion from Latin1 (ISO-8859-1) to Unicode because first 256 Unicode code points matches 8bit ISO-8859-1 codepage table. So when iocharset was specified and codepage not, then codepage used implicit value "iso8859-1".
So when hsb->nls_disk is not set and hsb->nls_io is then explicitly set hsb->nls_disk to "iso8859-1".
Such setup is obviously incompatible with Mac OS systems as they do not support iso8859-1 encoding for hfs. So print warning into dmesg about this fact.
After this change hsb->nls_disk is always set, so remove code paths for case when hsb->nls_disk was not set as they are not needed anymore.
--- fs/hfs/super.c | 31 +++++++++++++++++++++++++++++++ fs/hfs/trans.c | 38 ++++++++++++++------------------------ 2 files changed, 45 insertions(+), 24 deletions(-)
fs/hfs/super.c +31 -0
@@ -351,6 +351,37 @@ static int parse_options(char *options, struct hfs_sb_info *hsb) } } +if (hsb->nls_io && !hsb->nls_disk) { +/* +* Previous version of hfs driver did something unexpected: +* When codepage was not defined but iocharset was then +* hfs driver copied 8bit character from disk directly to +* 16bit unicode wchar_t type. Which means it did conversion +* from Latin1 (ISO-8859-1) to Unicode because first 256 +* Unicode code points matches 8bit ISO-8859-1 codepage table. +* So when iocharset was specified and codepage not, then +* codepage used implicit value "iso8859-1". +* +* To not change this previous default behavior as some users +* may depend on it, we load iso8859-1 NLS table explicitly +* to simplify code and make it more reable what happens. +* +* In context of hfs driver it is really strange to use +* ISO-8859-1 codepage table for storing data to disk, but +* nothing forbids it. Just it is highly incompatible with +* Mac OS systems. So via pr_warn() inform user that this +* is not probably what he wants. +*/ +pr_warn ("iocharset was specified but codepage not, " +"using default codepage=iso8859-1\n" ); +pr_warn ("this default codepage=iso8859-1 is incompatible with " +"Mac OS systems and may be changed in the future" ); + hsb->nls_disk =load_nls ("iso8859-1" ); +if (!hsb->nls_disk) { +pr_err ("unable to load iso8859-1 codepage\n" ); +return 0 ; + } + }if (hsb->nls_disk && !hsb->nls_io) { hsb->nls_io =load_nls_default ();if (!hsb->nls_io) {
fs/hfs/trans.c +14 -24
@@ -48,18 +48,13 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in)wchar_t ch;while (srclen >0 ) { -if (nls_disk) { - size = nls_disk->char2uni (src, srclen, &ch); -if (size <=0 ) { - ch ='?' ; - size =1 ; - } - src += size; - srclen -= size; - }else { - ch = *src++; - srclen--; + size = nls_disk->char2uni (src, srclen, &ch); +if (size <=0 ) { + ch ='?' ; + size =1 ; } + src += size; + srclen -= size;if (ch =='/' ) ch =':' ; size = nls_io->uni2char (ch, dst, dstlen);
@@ -119,20 +114,15 @@ void hfs_asc2mac(struct super_block *sb, struct hfs_name *out, const struct qstr srclen -= size;if (ch ==':' ) ch ='/' ; -if (nls_disk) { - size = nls_disk->uni2char (ch, dst, dstlen); -if (size <0 ) { -if (size == -ENAMETOOLONG ) - goto out; - *dst ='?' ; - size =1 ; - } - dst += size; - dstlen -= size; - }else { - *dst++ = ch >0xff ?'?' : ch; - dstlen--; + size = nls_disk->uni2char (ch, dst, dstlen); +if (size <0 ) { +if (size == -ENAMETOOLONG ) + goto out; + *dst ='?' ; + size =1 ; } + dst += size; + dstlen -= size; } }else {char ch;
Signature · 1 line
2.20.1
RFC PATCH v2 10/18 hfs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
In reply to Pali RohárNLS table for utf8 is broken and cannot be fixed.
So instead of broken utf8 nls functions char2uni() and uni2char() use functions utf8_to_utf32() and utf32_to_utf8() which implements correct encoding and decoding between Unicode code points and UTF-8 sequence.
When iochatset=utf8 is used then set hsb->nls_io to NULL and use it for distinguish between the fact if NLS table or native UTF-8 functions should be used.
--- fs/hfs/super.c | 33 ++++++++++++++++++++++----------- fs/hfs/trans.c | 24 ++++++++++++++++++++---- 2 files changed, 42 insertions(+), 15 deletions(-)
fs/hfs/super.c +22 -11
@@ -149,10 +149,13 @@ static int hfs_show_options(struct seq_file *seq, struct dentry *root)seq_printf (seq,",part=%u" , sbi->part);if (sbi->session >=0 )seq_printf (seq,",session=%u" , sbi->session); -if (sbi->nls_disk) +if (sbi->nls_disk) {seq_printf (seq,",codepage=%s" , sbi->nls_disk->charset); -if (sbi->nls_io) -seq_printf (seq,",iocharset=%s" , sbi->nls_io->charset); +if (sbi->nls_io) +seq_printf (seq,",iocharset=%s" , sbi->nls_io->charset); +else +seq_puts (seq,",iocharset=utf8" ); + }if (sbi->s_quiet)seq_printf (seq,",quiet" );return 0 ;
@@ -225,6 +228,7 @@ static int parse_options(char *options, struct hfs_sb_info *hsb)char *p;substring_t args[MAX_OPT_ARGS ];int tmp, token; +bool have_iocharset;/* initialize the sb with defaults */ hsb->s_uid =current_uid ();
@@ -239,6 +243,8 @@ static int parse_options(char *options, struct hfs_sb_info *hsb)if (!options)return 1 ; + have_iocharset = false; +while ((p =strsep (&options,"," )) !=NULL ) {if (!*p)continue ;
@@ -332,18 +338,22 @@ static int parse_options(char *options, struct hfs_sb_info *hsb)kfree (p);break ;case opt_iocharset: -if (hsb->nls_io) { +if (have_iocharset) {pr_err ("unable to change iocharset\n" );return 0 ; } p =match_strdup (&args[0 ]); -if (p) - hsb->nls_io =load_nls (p); -if (!hsb->nls_io) { -pr_err ("unable to load iocharset \"%s\"\n" , p); -kfree (p); +if (!p)return 0 ; +if (strcmp (p,"utf8" ) !=0 ) { + hsb->nls_io =load_nls (p); +if (!hsb->nls_io) { +pr_err ("unable to load iocharset \"%s\"\n" , p); +kfree (p); +return 0 ; + } } + have_iocharset = true;kfree (p);break ;default :
@@ -351,7 +361,7 @@ static int parse_options(char *options, struct hfs_sb_info *hsb) } } -if (hsb->nls_io && !hsb->nls_disk) { +if (have_iocharset && !hsb->nls_disk) {/* * Previous version of hfs driver did something unexpected: * When codepage was not defined but iocharset was then
@@ -382,7 +392,8 @@ static int parse_options(char *options, struct hfs_sb_info *hsb)return 0 ; } } -if (hsb->nls_disk && !hsb->nls_io) { +if (hsb->nls_disk && + !have_iocharset &&strcmp (CONFIG_NLS_DEFAULT ,"utf8" ) !=0 ) { hsb->nls_io =load_nls_default ();if (!hsb->nls_io) {pr_err ("unable to load default iocharset\n" );
fs/hfs/trans.c +20 -4
@@ -44,7 +44,7 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in) srclen =HFS_NAMELEN ; dst = out; dstlen =HFS_MAX_NAMELEN ; -if (nls_io) { +if (nls_disk) {wchar_t ch;while (srclen >0 ) {
@@ -57,7 +57,12 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in) srclen -= size;if (ch =='/' ) ch =':' ; - size = nls_io->uni2char (ch, dst, dstlen); +if (nls_io) + size = nls_io->uni2char (ch, dst, dstlen); +else if (dstlen >0 ) + size =utf32_to_utf8 (ch, dst, dstlen); +else + size = -ENAMETOOLONG ;if (size <0 ) {if (size == -ENAMETOOLONG ) goto out;
@@ -101,11 +106,22 @@ void hfs_asc2mac(struct super_block *sb, struct hfs_name *out, const struct qstr srclen = in->len; dst = out->name; dstlen =HFS_NAMELEN ; -if (nls_io) { +if (nls_disk) {wchar_t ch; +unicode_t u;while (srclen >0 && dstlen >0 ) { - size = nls_io->char2uni (src, srclen, &ch); +if (nls_io) + size = nls_io->char2uni (src, srclen, &ch); +else { + size =utf8_to_utf32 (src, srclen, &u); +if (size >=0 ) { +if (u <=MAX_WCHAR_T ) + ch = u; +else + size = -EINVAL ; + } + }if (size <0 ) { ch ='?' ; size =1 ;
Signature · 1 line
2.20.1
RFC PATCH v2 11/18 hfsplus: Do not use broken utf8 NLS table for iocharset=utf8 mount option
In reply to Pali RohárNLS table for utf8 is broken and cannot be fixed.
So instead of broken utf8 nls functions char2uni() and uni2char() use functions utf8_to_utf32() and utf32_to_utf8() which implements correct encoding and decoding between Unicode code points and UTF-8 sequence.
Note that this fs driver does not support full Unicode range, specially UTF-16 surrogate pairs are unsupported. This patch does not change this limitation and support for UTF-16 surrogate pairs stay unimplemented.
When iochatset=utf8 is used then set sbi->nls to NULL and use it for distinguish between the fact if NLS table or native UTF-8 functions should be used.
--- fs/hfsplus/dir.c | 7 +++++-- fs/hfsplus/options.c | 32 ++++++++++++++++++-------------- fs/hfsplus/super.c | 7 +------ fs/hfsplus/unicode.c | 31 ++++++++++++++++++++++++++++--- fs/hfsplus/xattr.c | 20 +++++++++++++------- fs/hfsplus/xattr_security.c | 6 ++++-- 6 files changed, 69 insertions(+), 34 deletions(-)
fs/hfsplus/dir.c +5 -2
@@ -144,7 +144,9 @@ static int hfsplus_readdir(struct file *file, struct dir_context *ctx) err =hfs_find_init (HFSPLUS_SB (sb)->cat_tree, &fd);if (err)return err; - strbuf =kmalloc (NLS_MAX_CHARSET_SIZE *HFSPLUS_MAX_STRLEN +1 ,GFP_KERNEL ); + len = (HFSPLUS_SB (sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_MAX_STRLEN +1 ; + strbuf =kmalloc (len,GFP_KERNEL );if (!strbuf) { err = -ENOMEM ; goto out;
@@ -203,7 +205,8 @@ static int hfsplus_readdir(struct file *file, struct dir_context *ctx)hfs_bnode_read (fd.bnode, &entry, fd.entryoffset, fd.entrylength); type =be16_to_cpu (entry.type); - len =NLS_MAX_CHARSET_SIZE *HFSPLUS_MAX_STRLEN ; + len = (HFSPLUS_SB (sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_MAX_STRLEN ; err =hfsplus_uni2asc (sb, &fd.key->cat.name, strbuf, &len);if (err) goto out;
fs/hfsplus/options.c +18 -14
@@ -104,6 +104,9 @@ int hfsplus_parse_options(char *input, struct hfsplus_sb_info *sbi)char *p;substring_t args[MAX_OPT_ARGS ];int tmp, token; +bool have_iocharset; + + have_iocharset = false;if (!input) goto done;
@@ -175,20 +178,24 @@ int hfsplus_parse_options(char *input, struct hfsplus_sb_info *sbi)pr_warn ("option nls= is deprecated, use iocharset=\n" ); fallthrough;case opt_iocharset: -if (sbi->nls) { +if (have_iocharset) {pr_err ("unable to change nls mapping\n" );return 0 ; } p =match_strdup (&args[0 ]); -if (p) - sbi->nls =load_nls (p); -if (!sbi->nls) { -pr_err ("unable to load nls mapping \"%s\"\n" , - p); -kfree (p); +if (!p)return 0 ; +if (strcmp (p,"utf8" ) !=0 ) { + sbi->nls =load_nls (p); +if (!sbi->nls) { +pr_err ("unable to load nls mapping " +"\"%s\"\n" , p); +kfree (p); +return 0 ; + } }kfree (p); + have_iocharset = true;break ;case opt_decompose:clear_bit (HFSPLUS_SB_NODECOMPOSE , &sbi->flags);
@@ -211,13 +218,10 @@ int hfsplus_parse_options(char *input, struct hfsplus_sb_info *sbi) } done: -if (!sbi->nls) { -/* try utf8 first, as this is the old default behaviour */ - sbi->nls =load_nls ("utf8" ); -if (!sbi->nls) - sbi->nls =load_nls_default (); -if (!sbi->nls) -return 0 ; +if (!have_iocharset) { +/* use utf8, as this is the old default behaviour */ +pr_debug ("using native UTF-8 without nls\n" ); +/* no sbi->nls means that native UTF-8 code is used */ }return 1 ;
fs/hfsplus/super.c +1 -6
@@ -403,11 +403,7 @@ static int hfsplus_fill_super(struct super_block *sb, void *data, int silent)/* temporarily use utf8 to correctly find the hidden dir below */ nls = sbi->nls; - sbi->nls =load_nls ("utf8" ); -if (!sbi->nls) { -pr_err ("unable to load nls for utf8\n" ); - goto out_unload_nls; - } + sbi->nls =NULL ;/* Grab the volume header */ if (hfsplus_read_wrapper (sb)) {
@@ -585,7 +581,6 @@ static int hfsplus_fill_super(struct super_block *sb, void *data, int silent) } } -unload_nls (sbi->nls); sbi->nls = nls;return 0 ;
fs/hfsplus/unicode.c +28 -3
@@ -190,7 +190,12 @@ int hfsplus_uni2asc(struct super_block *sb, c0 =':' ;break ; } - res = nls->uni2char (c0, op, len); +if (nls) + res = nls->uni2char (c0, op, len); +else if (len >0 ) + res =utf32_to_utf8 (c0, op, len); +else + res = -ENAMETOOLONG ;if (res <0 ) {if (res == -ENAMETOOLONG ) goto out;
@@ -233,7 +238,12 @@ int hfsplus_uni2asc(struct super_block *sb, cc = c0; } done: - res = nls->uni2char (cc, op, len); +if (nls) + res = nls->uni2char (cc, op, len); +else if (len >0 ) + res =utf32_to_utf8 (cc, op, len); +else + res = -ENAMETOOLONG ;if (res <0 ) {if (res == -ENAMETOOLONG ) goto out;
@@ -256,7 +266,22 @@ int hfsplus_uni2asc(struct super_block *sb,static inline int asc2unichar (struct super_block *sb,const char *astr,int len,wchar_t *uc) { -int size =HFSPLUS_SB (sb)->nls->char2uni (astr, len, uc); +struct nls_table *nls =HFSPLUS_SB (sb)->nls; +unicode_t u; +int size; + +if (nls) + size = nls->char2uni (astr, len, uc); +else { + size =utf8_to_utf32 (astr, len, &u); +if (size >=0 ) { +/* TODO: Add support for UTF-16 surrogate pairs */ +if (u <=MAX_WCHAR_T ) + *uc = u; +else + size = -EINVAL ; + } + }if (size <=0 ) { *uc ='?' ; size =1 ;
fs/hfsplus/xattr.c +13 -7
@@ -422,11 +422,13 @@ int hfsplus_setxattr(struct inode *inode, const char *name, const void *value, size_t size, int flags, const char *prefix, size_t prefixlen) { + int xattr_name_len;char *xattr_name;int res; - xattr_name =kmalloc (NLS_MAX_CHARSET_SIZE *HFSPLUS_ATTR_MAX_STRLEN +1 , -GFP_KERNEL ); + xattr_name_len = (HFSPLUS_SB (inode->i_sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_ATTR_MAX_STRLEN +1 ; + xattr_name =kmalloc (xattr_name_len,GFP_KERNEL );if (!xattr_name)return -ENOMEM ;strcpy (xattr_name, prefix);
@@ -578,9 +580,11 @@ ssize_t hfsplus_getxattr(struct inode *inode, const char *name, {int res;char *xattr_name; +int xattr_name_len; - xattr_name =kmalloc (NLS_MAX_CHARSET_SIZE *HFSPLUS_ATTR_MAX_STRLEN +1 , -GFP_KERNEL ); + xattr_name_len = (HFSPLUS_SB (inode->i_sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_ATTR_MAX_STRLEN +1 ; + xattr_name =kmalloc (xattr_name_len,GFP_KERNEL );if (!xattr_name)return -ENOMEM ;
@@ -699,8 +703,9 @@ ssize_t hfsplus_listxattr(struct dentry *dentry, char *buffer, size_t size)return err; } - strbuf =kmalloc (NLS_MAX_CHARSET_SIZE *HFSPLUS_ATTR_MAX_STRLEN + -XATTR_MAC_OSX_PREFIX_LEN +1 ,GFP_KERNEL ); + xattr_name_len = (HFSPLUS_SB (inode->i_sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_ATTR_MAX_STRLEN +XATTR_MAC_OSX_PREFIX_LEN +1 ; + strbuf =kmalloc (xattr_name_len,GFP_KERNEL );if (!strbuf) { res = -ENOMEM ; goto out;
@@ -732,7 +737,8 @@ ssize_t hfsplus_listxattr(struct dentry *dentry, char *buffer, size_t size)if (be32_to_cpu (attr_key.cnid) != inode->i_ino) goto end_listxattr; - xattr_name_len =NLS_MAX_CHARSET_SIZE *HFSPLUS_ATTR_MAX_STRLEN ; + xattr_name_len = (HFSPLUS_SB (inode->i_sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_ATTR_MAX_STRLEN ;if (hfsplus_uni2asc (inode->i_sb, (const struct hfsplus_unistr *)&fd.key->attr.key_name, strbuf, &xattr_name_len)) {
fs/hfsplus/xattr_security.c +4 -2
@@ -38,11 +38,13 @@ static int hfsplus_initxattrs(struct inode *inode, void *fs_info) {const struct xattr *xattr; +int xattr_name_len;char *xattr_name;int err =0 ; - xattr_name =kmalloc (NLS_MAX_CHARSET_SIZE *HFSPLUS_ATTR_MAX_STRLEN +1 , -GFP_KERNEL ); + xattr_name_len = (HFSPLUS_SB (inode->i_sb)->nls ?NLS_MAX_CHARSET_SIZE :4 ) * +HFSPLUS_ATTR_MAX_STRLEN +1 ; + xattr_name =kmalloc (xattr_name_len,GFP_KERNEL );if (!xattr_name)return -ENOMEM ;for (xattr = xattr_array; xattr->name !=NULL ; xattr++) {
Signature · 1 line
2.20.1
RFC PATCH v2 12/18 jfs: Remove custom iso8859-1 implementation
In reply to Pali RohárWhen iocharset= mount option is not specified or when is set to iocharset=none then jfs driver uses its own custom iso8895-1 encoding implementation.
NLS already provides iso8895-1 module, so use it instead of custom jfs iso8859-1 implementation.
--- fs/jfs/jfs_unicode.c | 14 +------------- fs/jfs/super.c | 29 +++++++++++++++++++---------- 2 files changed, 20 insertions(+), 23 deletions(-)
fs/jfs/jfs_unicode.c +1 -13
@@ -33,13 +33,8 @@ int jfs_strfromUCS_le(char *to, const __le16 * from, NLS_MAX_CHARSET_SIZE);if (charlen >0 ) outlen += charlen; -else +else { to[outlen++] ='?' ; - } - }else { -for (i =0 ; (i < len) && from[i]; i++) { -if (unlikely (le16_to_cpu (from[i]) &0xff00 )) { - to[i] ='?' ;if (unlikely (warn)) { warn--; warn_again--;
@@ -49,12 +44,8 @@ int jfs_strfromUCS_le(char *to, const __le16 * from,printk (KERN_ERR "mount with iocharset=utf8 to access\n" ); } - } -else - to[i] = (char ) (le16_to_cpu (from[i])); } - outlen = i; } to[outlen] =0 ;return outlen;
@@ -84,9 +75,6 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len,return charlen; } } - }else { -for (i =0 ; (i < len) && from[i]; i++) - to[i] = (wchar_t ) from[i]; } to[i] =0 ;
fs/jfs/super.c +19 -10
@@ -231,7 +231,7 @@ static const match_table_t tokens = { };static int parse_options (char *options,struct super_block *sb,s64 *newLVSize, -int *flag) +int *flag,int remount) {void *nls_map = (void *)-1 ;/* -1: no change; NULL: none */ char *p;
@@ -263,14 +263,14 @@ static int parse_options(char *options, struct super_block *sb, s64 *newLVSize,case Opt_iocharset:if (nls_map && nls_map != (void *)-1 )unload_nls (nls_map); -if (!strcmp (args[0 ].from,"none" )) - nls_map =NULL ; -else { +/* compatibility alias none means ISO-8859-1 */ +if (strcmp (args[0 ].from,"none" ) ==0 ) + nls_map =load_nls ("iso8859-1" ); +else nls_map =load_nls (args[0 ].from); -if (!nls_map) { -pr_err ("JFS: charset not found\n" ); - goto cleanup; - } +if (!nls_map) { +pr_err ("JFS: charset not found\n" ); + goto cleanup; }break ;case Opt_resize:
@@ -409,6 +409,15 @@ static int parse_options(char *options, struct super_block *sb, s64 *newLVSize, } } +if (!remount && nls_map == (void *)-1 ) { +/* Previously default NLS table was ISO-8859-1 */ + nls_map =load_nls ("iso8859-1" ); +if (!nls_map) { +pr_err ("JFS: iso8859-1 charset not found\n" ); + goto cleanup; + } + } +if (nls_map != (void *)-1 ) {/* Discard old (if remount) */ unload_nls (sbi->nls_tab);
@@ -430,7 +439,7 @@ static int jfs_remount(struct super_block *sb, int *flags, char *data)int ret;sync_filesystem (sb); -if (!parse_options (data, sb, &newLVSize, &flag)) +if (!parse_options (data, sb, &newLVSize, &flag,1 ))return -EINVAL ;if (newLVSize) {
@@ -508,7 +517,7 @@ static int jfs_fill_super(struct super_block *sb, void *data, int silent)/* initialize the mount flag and determine the default error handler */ flag =JFS_ERR_REMOUNT_RO ; -if (!parse_options ((char *) data, sb, &newLVSize, &flag)) +if (!parse_options ((char *) data, sb, &newLVSize, &flag,0 )) goto out_kfree; sbi->flag = flag;
Signature · 1 line
2.20.1
RFC PATCH v2 13/18 jfs: Fix buffer overflow in jfs_strfromUCS_le() function
In reply to Pali RohárFunction jfs_strfromUCS_le() writes to unknown offset in buffer allocated by __get_free_page(GFP_KERNEL). So it cannot expects that there is least NLS_MAX_CHARSET_SIZE bytes space before end of that buffer.
Fix this issue by add a new parameter maxlen for jfs_strfromUCS_le() function. And use it for passing remaining size of buffer to prevent buffer overflow in kernel.
--- fs/jfs/jfs_dtree.c | 13 ++++++++++--- fs/jfs/jfs_unicode.c | 6 +++--- fs/jfs/jfs_unicode.h | 2 +- 3 files changed, 14 insertions(+), 7 deletions(-)
fs/jfs/jfs_dtree.c +10 -3
@@ -2715,6 +2715,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)int d_namleft, len, outlen;unsigned long dirent_buf;char *name_ptr; +int maxlen;u32 dir_index;int do_index =0 ;uint loop_count =0 ;
@@ -2937,7 +2938,10 @@ int jfs_readdir(struct file *file, struct dir_context *ctx) }/* copy the name of head/only segment */ - outlen =jfs_strfromUCS_le (name_ptr, d->name, len, + maxlen =PAGE_SIZE -sizeof (struct jfs_dirent ) - + (name_ptr - jfs_dirent->name); + outlen =jfs_strfromUCS_le (name_ptr, maxlen, + d->name, len, codepage); jfs_dirent->name_len = outlen;
@@ -2957,8 +2961,11 @@ int jfs_readdir(struct file *file, struct dir_context *ctx) goto skip_one; } len =min (d_namleft,DTSLOTDATALEN ); - outlen =jfs_strfromUCS_le (name_ptr, t->name, - len, codepage); + maxlen =PAGE_SIZE -sizeof (struct jfs_dirent ) - + (name_ptr - jfs_dirent->name); + outlen =jfs_strfromUCS_le (name_ptr, maxlen, + t->name, len, + codepage); jfs_dirent->name_len += outlen; next = t->next;
fs/jfs/jfs_unicode.c +3 -3
@@ -16,7 +16,7 @@* FUNCTION: Convert little-endian unicode string to character string * */ -int jfs_strfromUCS_le (char *to,const __le16 * from, +int jfs_strfromUCS_le (char *to,int maxlen,const __le16 * from,int len,struct nls_table *codepage) {int i;
@@ -25,12 +25,12 @@ int jfs_strfromUCS_le(char *to, const __le16 * from,int warn = !!warn_again;/* once per string */ if (codepage) { -for (i =0 ; (i < len) && from[i]; i++) { +for (i =0 ; (i < len) && from[i] && outlen < maxlen-1 ; i++) {int charlen; charlen = codepage->uni2char (le16_to_cpu (from[i]), &to[outlen], -NLS_MAX_CHARSET_SIZE ); + maxlen-1 -outlen);if (charlen >0 ) outlen += charlen;else {
fs/jfs/jfs_unicode.h +1 -1
@@ -19,7 +19,7 @@ typedef struct {extern signed char UniUpperTable[512 ];extern UNICASERANGE UniUpperRange[];extern int get_UCSname (struct component_name *,struct dentry *); -extern int jfs_strfromUCS_le (char *,const __le16 *,int ,struct nls_table *); +extern int jfs_strfromUCS_le (char *,int ,const __le16 *,int ,struct nls_table *);#define free_UCSname (COMP ) kfree((COMP)->name)
Signature · 1 line
2.20.1
RFC PATCH v2 14/18 jfs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
In reply to Pali RohárNLS table for utf8 is broken and cannot be fixed.
So instead of broken utf8 nls functions char2uni() and uni2char() use functions utf8s_to_utf16s() and utf16s_to_utf8s() which implements correct conversion between UTF-16 and UTF-8.
These functions implements also correct processing of UTF-16 surrogate pairs and therefore after this change jfs driver would be able to correctly handle also file names with 4-byte UTF-8 sequences.
When iochatset=utf8 is used then set sbi->nls_tab to NULL and use it for distinguish between the fact if NLS table or native UTF-8 functions should be used.
--- fs/jfs/jfs_unicode.c | 17 +++++++++++++++-- fs/jfs/super.c | 24 +++++++++++++++--------- 2 files changed, 30 insertions(+), 11 deletions(-)
fs/jfs/jfs_unicode.c +15 -2
@@ -46,6 +46,9 @@ int jfs_strfromUCS_le(char *to, int maxlen, const __le16 * from, } } } + }else { + outlen =utf16s_to_utf8s ((const wchar_t *)from, len, +UTF16_LITTLE_ENDIAN , to, maxlen-1 ); } to[outlen] =0 ;return outlen;
@@ -61,6 +64,7 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len, struct nls_table *codepage) {int charlen; +int outlen;int i;if (codepage) {
@@ -75,10 +79,19 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len,return charlen; } } + outlen = i; + }else { + outlen =utf8s_to_utf16s (from, len,UTF16_LITTLE_ENDIAN , + to, len); +if (outlen <1 ) { +jfs_err ("jfs_strtoUCS: utf8s_to_utf16s returned %d." , + outlen); +return outlen; + } } - to[i] =0 ; -return i; + to[outlen] =0 ; +return outlen; }/*
fs/jfs/super.c +15 -9
@@ -261,16 +261,20 @@ static int parse_options(char *options, struct super_block *sb, s64 *newLVSize,/* Don't do anything ;-) */ break ;case Opt_iocharset: -if (nls_map && nls_map != (void *)-1 ) +if (nls_map && nls_map != (void *)-1 ) {unload_nls (nls_map); -/* compatibility alias none means ISO-8859-1 */ -if (strcmp (args[0 ].from,"none" ) ==0 ) - nls_map =load_nls ("iso8859-1" ); -else - nls_map =load_nls (args[0 ].from); -if (!nls_map) { -pr_err ("JFS: charset not found\n" ); - goto cleanup; + nls_map =NULL ; + } +if (strcmp (args[0 ].from,"utf8" ) !=0 ) { +/* compatibility alias none means ISO-8859-1 */ +if (strcmp (args[0 ].from,"none" ) ==0 ) + nls_map =load_nls ("iso8859-1" ); +else + nls_map =load_nls (args[0 ].from); +if (!nls_map) { +pr_err ("JFS: charset not found\n" ); + goto cleanup; + } }break ;case Opt_resize:
@@ -713,6 +717,8 @@ static int jfs_show_options(struct seq_file *seq, struct dentry *root)seq_printf (seq,",discard=%u" , sbi->minblks_trim);if (sbi->nls_tab)seq_printf (seq,",iocharset=%s" , sbi->nls_tab->charset); +else +seq_puts (seq,",iocharset=utf8" );if (sbi->flag &JFS_ERR_CONTINUE )seq_printf (seq,",errors=continue" );if (sbi->flag &JFS_ERR_PANIC )
Signature · 1 line
2.20.1
RFC PATCH v2 15/18 ntfs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
In reply to Pali RohárNLS table for utf8 is broken and cannot be fixed.
So instead of broken utf8 nls functions char2uni() and uni2char() use functions utf8s_to_utf16s() and utf16s_to_utf8s() which implements correct conversion between UTF-16 and UTF-8.
These functions implements also correct processing of UTF-16 surrogate pairs and therefore after this change ntfs driver would be able to correctly handle also file names with 4-byte UTF-8 sequences.
When iochatset=utf8 is used then set vol->nls_map to NULL and use it for distinguish between the fact if NLS table or native UTF-8 functions should be used.
--- fs/ntfs/dir.c | 6 ++++-- fs/ntfs/inode.c | 5 ++++- fs/ntfs/super.c | 41 ++++++++++++++++++++++++----------------- fs/ntfs/unistr.c | 28 +++++++++++++++++++++++++--- 4 files changed, 57 insertions(+), 23 deletions(-)
fs/ntfs/dir.c +4 -2
@@ -1034,7 +1034,8 @@ static inline int ntfs_filldir(ntfs_volume *vol, } name_len =ntfs_ucstonls (vol, (ntfschar *)&ie->key.file_name.file_name, ie->key.file_name.file_name_length, &name, -NTFS_MAX_NAME_LEN *NLS_MAX_CHARSET_SIZE +1 ); +NTFS_MAX_NAME_LEN * + (vol->nls_map ?NLS_MAX_CHARSET_SIZE :4 ) +1 );if (name_len <=0 ) {ntfs_warning (vol->sb,"Skipping unrepresentable inode 0x%llx." , (long long )MREF_LE (ie->data.dir.indexed_file));
@@ -1118,7 +1119,8 @@ static int ntfs_readdir(struct file *file, struct dir_context *actor)* Allocate a buffer to store the current name being processed * converted to format determined by current NLS. */ - name =kmalloc (NTFS_MAX_NAME_LEN *NLS_MAX_CHARSET_SIZE +1 ,GFP_NOFS ); + name =kmalloc (NTFS_MAX_NAME_LEN * + (vol->nls_map ?NLS_MAX_CHARSET_SIZE :4 ) +1 ,GFP_NOFS );if (unlikely (!name)) { err = -ENOMEM ; goto err_out;
fs/ntfs/inode.c +4 -1
@@ -2314,7 +2314,10 @@ int ntfs_show_options(struct seq_file *sf, struct dentry *root)seq_printf (sf,",fmask=0%o" , vol->fmask);seq_printf (sf,",dmask=0%o" , vol->dmask); } -seq_printf (sf,",iocharset=%s" , vol->nls_map->charset); +if (vol->nls_map) +seq_printf (sf,",iocharset=%s" , vol->nls_map->charset); +else +seq_puts (sf,",iocharset=utf8" );if (NVolCaseSensitive (vol))seq_printf (sf,",case_sensitive" );if (NVolShowSystemFiles (vol))
fs/ntfs/super.c +24 -17
88 lines not on this page · raw message
fs/ntfs/unistr.c +25 -3
@@ -254,6 +254,17 @@ int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,if (likely (ins)) { ucs =kmem_cache_alloc (ntfs_name_cache,GFP_NOFS );if (likely (ucs)) { +if (!nls) { + wc_len =utf8s_to_utf16s (ins, ins_len, +UTF16_LITTLE_ENDIAN , + (wchar_t *)ucs, +NTFS_MAX_NAME_LEN ); +if (wc_len <0 || wc_len >=NTFS_MAX_NAME_LEN ) + goto name_err; + ucs[wc_len] =0 ; + *outs = ucs; +return o; + }for (i = o =0 ; i < ins_len; i += wc_len) { wc_len = nls->char2uni (ins + i, ins_len - i, &wc);
@@ -283,7 +294,7 @@ int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,if (wc_len <0 ) {ntfs_error (vol->sb,"Name using character set %s contains " "characters that cannot be converted to " -"Unicode." , nls->charset); +"Unicode." , nls ? nls->charset :"utf8" ); i = -EILSEQ ; }else /* if (o >= NTFS_MAX_NAME_LEN) */ { ntfs_error(vol->sb, "Name is too long (maximum length for a "
@@ -335,11 +346,22 @@ int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins, goto conversion_err; }if (!ns) { - ns_len = ins_len *NLS_MAX_CHARSET_SIZE ; + ns_len = ins_len * (nls ?NLS_MAX_CHARSET_SIZE :4 ); ns =kmalloc (ns_len +1 ,GFP_NOFS );if (!ns) goto mem_err_out; } +if (!nls) { + o =utf16s_to_utf8s ((const wchar_t *)ins, ins_len, +UTF16_LITTLE_ENDIAN , ns, ns_len); +if (o >= ns_len) { + wc = -ENAMETOOLONG ; + goto conversion_err; + } + ns[o] =0 ; + *outs = ns; +return o; + }for (i = o =0 ; i < ins_len; i++) { retry: wc = nls->uni2char (le16_to_cpu (ins[i]), ns + o, ns_len - o);
@@ -373,7 +395,7 @@ retry: wc = nls->uni2char(le16_to_cpu(ins[i]), ns + o,ntfs_error (vol->sb,"Unicode name contains characters that cannot be " "converted to character set %s. You might want to " "try to use the mount option iocharset=utf8." , - nls->charset); + nls ? nls->charset :"utf8" );if (ns != *outs)kfree (ns);if (wc != -ENAMETOOLONG )
Signature · 1 line
2.20.1
RFC PATCH v2 17/18 cifs: Remove usage of load_nls_default() calls
In reply to Pali Rohárcifs functions will use UTF-8 encoding when nls_table is set to NULL. So there is no need to load "dummy" NLS table for some operations.
On few places in dfs_cache replace utf8 nls by utf8_to_utf32() function which converts UTF-8 sequence to unicode code points (stored as type unicode_t). This should fix handling of (UTF-16) CIFS paths with UTF-16 surrogare pairs, which utf8 nls module cannot handle.
--- fs/cifs/cifssmb.c | 8 ++------ fs/cifs/dfs_cache.c | 24 ++++++++---------------- fs/cifs/smb2pdu.c | 18 ++++-------------- 3 files changed, 14 insertions(+), 36 deletions(-)
fs/cifs/cifssmb.c +2 -6
@@ -69,7 +69,6 @@ cifs_reconnect_tcon(struct cifs_tcon *tcon, int smb_command)int rc;struct cifs_ses *ses;struct TCP_Server_Info *server; -struct nls_table *nls_codepage;int retries;/*
@@ -147,8 +146,6 @@ cifs_reconnect_tcon(struct cifs_tcon *tcon, int smb_command) }spin_unlock (&ses->chan_lock); - nls_codepage =load_nls_default (); -/* * Recheck after acquire mutex. If another thread is negotiating * and the server never sends an answer the socket will be closed
@@ -182,7 +179,7 @@ cifs_reconnect_tcon(struct cifs_tcon *tcon, int smb_command)mutex_lock (&ses->session_mutex); rc =cifs_negotiate_protocol (0 , ses, server);if (!rc) - rc =cifs_setup_session (0 , ses, server, nls_codepage); + rc =cifs_setup_session (0 , ses, server,NULL );/* do we need to reconnect tcon? */ if (rc || !tcon->need_reconnect) {
@@ -192,7 +189,7 @@ cifs_reconnect_tcon(struct cifs_tcon *tcon, int smb_command) skip_sess_setup:cifs_mark_open_files_invalid (tcon); - rc =cifs_tree_connect (0 , tcon, nls_codepage); + rc =cifs_tree_connect (0 , tcon,NULL );mutex_unlock (&ses->session_mutex);cifs_dbg (FYI ,"reconnect tcon rc = %d\n" , rc);
@@ -228,7 +225,6 @@ cifs_reconnect_tcon(struct cifs_tcon *tcon, int smb_command) rc = -EAGAIN ; } -unload_nls (nls_codepage);return rc; }
fs/cifs/dfs_cache.c +8 -16
@@ -66,8 +66,6 @@ static struct workqueue_struct *dfscache_wq __read_mostly;static int cache_ttl;static DEFINE_SPINLOCK (cache_ttl_lock); -static struct nls_table *cache_cp; -/* * Number of entries in the cache */
@@ -173,14 +171,14 @@ char *dfs_cache_canonical_path(const char *path, const struct nls_table *cp, intif (!path ||strlen (path) <3 || (*path !='\\' && *path !='/' ))return ERR_PTR (-EINVAL ); -if (unlikely (strcmp (cp->charset, cache_cp->charset))) { +if (unlikely (cp)) { tmp = (char *)cifs_strndup_to_utf16 (path,strlen (path), &plen, cp, remap);if (!tmp) {cifs_dbg (VFS ,"%s: failed to convert path to utf16\n" , __func__);return ERR_PTR (-EINVAL ); } - npath =cifs_strndup_from_utf16 (tmp, plen, true, cache_cp); + npath =cifs_strndup_from_utf16 (tmp, plen, true,NULL );kfree (tmp);if (!npath) {
@@ -392,9 +390,6 @@ int dfs_cache_init(void)INIT_HLIST_HEAD (&cache_htable[i]);atomic_set (&cache_count,0 ); - cache_cp =load_nls ("utf8" ); -if (!cache_cp) - cache_cp =load_nls_default ();cifs_dbg (FYI ,"%s: initialized DFS referral cache\n" , __func__);return 0 ;
@@ -408,11 +403,11 @@ static int cache_entry_hash(const void *data, int size, unsigned int *hash) {int i, clen;const unsigned char *s = data; -wchar_t c; +unicode_t c;unsigned int h =0 ;for (i =0 ; i < size; i += clen) { - clen = cache_cp->char2uni (&s[i], size - i, &c); + clen =utf8_to_utf32 (&s[i], size - i, &c);if (unlikely (clen <0 )) {cifs_dbg (VFS ,"%s: can't convert char\n" , __func__);return clen;
@@ -601,14 +596,14 @@ static int add_cache_entry_locked(struct dfs_info3_param *refs, int numrefs)static bool dfs_path_equal (const char *s1,int len1,const char *s2,int len2) {int i, l1, l2; -wchar_t c1, c2; +unicode_t c1, c2;if (len1 != len2)return false;for (i =0 ; i < len1; i += l1) { - l1 = cache_cp->char2uni (&s1[i], len1 - i, &c1); - l2 = cache_cp->char2uni (&s2[i], len2 - i, &c2); + l1 =utf8_to_utf32 (&s1[i], len1 - i, &c1); + l2 =utf8_to_utf32 (&s2[i], len2 - i, &c2);if (unlikely (l1 <0 && l2 <0 )) {if (s1[i] != s2[i])return false;
@@ -698,7 +693,6 @@ static struct cache_entry *lookup_cache_entry(const char *path)void dfs_cache_destroy (void ) {cancel_delayed_work_sync (&refresh_task); -unload_nls (cache_cp);free_mount_group_list ();flush_cache_ents ();kmem_cache_destroy (cache_slab);
@@ -744,11 +738,9 @@ static int get_dfs_referral(const unsigned int xid, struct cifs_ses *ses, constif (!ses || !ses->server || !ses->server->ops->get_dfs_refer)return -EOPNOTSUPP ; -if (unlikely (!cache_cp)) -return -EINVAL ;cifs_dbg (FYI ,"%s: ipc=%s referral=%s\n" , __func__, ses->tcon_ipc->tree_name, path); - rc = ses->server->ops->get_dfs_refer (xid, ses, path, refs, numrefs, cache_cp, + rc = ses->server->ops->get_dfs_refer (xid, ses, path, refs, numrefs,NULL ,NO_MAP_UNI_RSVD );if (!rc) {struct dfs_info3_param *ref = *refs;
fs/cifs/smb2pdu.c +4 -14
@@ -144,7 +144,6 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon, struct TCP_Server_Info *server) {int rc =0 ; -struct nls_table *nls_codepage;struct cifs_ses *ses;int retries;
@@ -250,8 +249,6 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon, tcon->ses->chans_need_reconnect, tcon->need_reconnect); - nls_codepage =load_nls_default (); -/* * Recheck after acquire mutex. If another thread is negotiating * and the server never sends an answer the socket will be closed
@@ -284,7 +281,7 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon,mutex_lock (&ses->session_mutex); rc =cifs_negotiate_protocol (0 , ses, server);if (!rc) { - rc =cifs_setup_session (0 , ses, server, nls_codepage); + rc =cifs_setup_session (0 , ses, server,NULL );if ((rc == -EACCES ) && !tcon->retry) {mutex_unlock (&ses->session_mutex); rc = -EHOSTDOWN ;
@@ -309,7 +306,7 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon,if (tcon->use_persistent) tcon->need_reopen_files = true; - rc =cifs_tree_connect (0 , tcon, nls_codepage); + rc =cifs_tree_connect (0 , tcon,NULL );mutex_unlock (&ses->session_mutex);cifs_dbg (FYI ,"reconnect tcon rc = %d\n" , rc);
@@ -345,7 +342,6 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon, rc = -EAGAIN ; } failed: -unload_nls (nls_codepage);return rc; }
@@ -503,12 +499,10 @@ build_encrypt_ctxt(struct smb2_encryption_neg_context *pneg_ctxt)static unsigned int build_netname_ctxt (struct smb2_netname_neg_context *pneg_ctxt,char *hostname) { -struct nls_table *cp =load_nls_default (); - pneg_ctxt->ContextType =SMB2_NETNAME_NEGOTIATE_CONTEXT_ID ;/* copy up to max of first 100 bytes of server name to NetName field */ - pneg_ctxt->DataLength =cpu_to_le16 (2 *cifs_strtoUTF16 (pneg_ctxt->NetName, hostname,100 , cp)); + pneg_ctxt->DataLength =cpu_to_le16 (2 *cifs_strtoUTF16 (pneg_ctxt->NetName, hostname,100 ,NULL ));/* context size is DataLength + minimal smb2_neg_context */ return ALIGN (le16_to_cpu (pneg_ctxt->DataLength) +sizeof (struct smb2_neg_context ),8 ); }
@@ -2568,7 +2562,6 @@ alloc_path_with_tree_prefix(__le16 **out_path, int *out_size, int *out_len, const char *treename, const __le16 *path) { int treename_len, path_len; -struct nls_table *cp;const __le16 sep[] = {cpu_to_le16 ('\\' ),cpu_to_le16 (0x0000 )};/*
@@ -2595,8 +2588,7 @@ alloc_path_with_tree_prefix(__le16 **out_path, int *out_size, int *out_len,if (!*out_path)return -ENOMEM ; - cp =load_nls_default (); -cifs_strtoUTF16 (*out_path, treename, treename_len, cp); +cifs_strtoUTF16 (*out_path, treename, treename_len,NULL );/* Do not append the separator if the path is empty */ if (path[0 ] !=cpu_to_le16 (0x0000 )) {
@@ -2604,8 +2596,6 @@ alloc_path_with_tree_prefix(__le16 **out_path, int *out_size, int *out_len,UniStrcat (*out_path, path); } -unload_nls (cp); -return 0 ; }
Signature · 1 line
2.20.1
RFC PATCH v2 16/18 cifs: Do not use broken utf8 NLS table for iocharset=utf8 mount option
In reply to Pali RohárNLS table for utf8 is broken and cannot be fixed.
So instead of broken utf8 nls functions char2uni() and uni2char() use functions utf8s_to_utf16s() and utf16s_to_utf8s() which implements correct conversion between UTF-16 and UTF-8.
When iochatset=utf8 is used then set ctx->iocharset to NULL and use it for distinguish between the fact if NLS table or native UTF-8 functions should be used.
--- fs/cifs/cifs_unicode.c | 128 +++++++++++++++++++++++++++-------------- fs/cifs/cifs_unicode.h | 2 +- fs/cifs/cifsfs.c | 2 + fs/cifs/connect.c | 8 ++- fs/cifs/dir.c | 28 +++++++-- fs/cifs/winucase.c | 14 +++-- 6 files changed, 124 insertions(+), 58 deletions(-)
fs/cifs/cifs_unicode.c +84 -44
@@ -130,20 +130,17 @@ cifs_mapchar(char *target, const __u16 *from, const struct nls_table *cp, convert_sfu_char(src_char, target))return len; -/* if character not one of seven in special remap set */ - len = cp->uni2char (src_char, target,NLS_MAX_CHARSET_SIZE ); -if (len <=0 ) - goto surrogate_pair; - -return len; +if (cp) { +/* if character not one of seven in special remap set */ + len = cp->uni2char (src_char, target,NLS_MAX_CHARSET_SIZE ); +if (len <=0 ) + goto unknown; + }else { + len =utf16s_to_utf8s (from,3 ,UTF16_LITTLE_ENDIAN , target,6 ); +if (len <=0 ) + goto unknown; + } -surrogate_pair: -/* convert SURROGATE_PAIR and IVS */ -if (strcmp (cp->charset,"utf8" )) - goto unknown; - len =utf16s_to_utf8s (from,3 ,UTF16_LITTLE_ENDIAN , target,6 ); -if (len <=0 ) - goto unknown;return len; unknown:
@@ -239,6 +236,37 @@ cifs_from_utf16(char *to, const __le16 *from, int tolen, int fromlen,return outlen; } +static int cifs_utf8s_to_utf16s (const char *s,int inlen,__le16 *pwcs) +{ +__le16 *op; +int size; +unicode_t u; + + op = pwcs; +while (inlen >0 && *s) { +if (*s &0x80 ) { + size =utf8_to_utf32 (s, inlen, &u); +if (size <=0 ) { + u =0x003f ;/* A question mark */ + size =1 ; + } + s += size; + inlen -= size; +if (u >=0x10000 ) { + u -=0x10000 ; + *op++ =__cpu_to_le16 (0xd800 | ((u >>10 ) &0x03ff )); + *op++ =__cpu_to_le16 (0xdc00 | (u &0x03ff )); + }else { + *op++ =__cpu_to_le16 (u); + } + }else { + *op++ =__cpu_to_le16 (*s++); + inlen--; + } + } +return op - pwcs; +} +/* * NAME: cifs_strtoUTF16() *
@@ -254,24 +282,14 @@ cifs_strtoUTF16(__le16 *to, const char *from, int len,wchar_t wchar_to;/* needed to quiet sparse */ /* special case for utf8 to handle no plane0 chars */ -if (!strcmp (codepage->charset,"utf8" )) { +if (!codepage) {/* * convert utf8 -> utf16, we assume we have enough space * as caller should have assumed conversion does not overflow -* in destination len is length in wchar_t units (16bits) -*/ - i =utf8s_to_utf16s (from, len,UTF16_LITTLE_ENDIAN , - (wchar_t *) to, len); - -/* if success terminate and exit */ -if (i >=0 ) - goto success; -/* -* if fails fall back to UCS encoding as this -* function should not return negative values -* currently can fail only if source contains -* invalid encoded characters +* in destination len is length in __le16 units */ + i =cifs_utf8s_to_utf16s (from, len, to); + goto success; }for (i =0 ; len && *from; i++, from += charlen, len -= charlen) {
@@ -502,25 +520,29 @@ cifsConvertToUTF16(__le16 *target, const char *source, int srclen,* as they use backslash as separator. */ if (dst_char ==0 ) { - charlen = cp->char2uni (source + i, srclen - i, &tmp); - dst_char =cpu_to_le16 (tmp); - -/* -* if no match, use question mark, which at least in -* some cases serves as wild card -*/ -if (charlen >0 ) - goto ctoUTF16; - -/* convert SURROGATE_PAIR */ -if (strcmp (cp->charset,"utf8" ) || !wchar_to) - goto unknown; -if (*(source + i) &0x80 ) { - charlen =utf8_to_utf32 (source + i,6 , &u); -if (charlen <0 ) +if (cp) { + charlen = cp->char2uni (source + i, srclen - i, &tmp); + dst_char =cpu_to_le16 (tmp); + +/* +* if no match, use question mark, which at least in +* some cases serves as wild card +*/ +if (charlen >0 ) + goto ctoUTF16; +else goto unknown; - }else + } + +/* UTF-8 to UTF-16 conversion */ + +if (!wchar_to) goto unknown; + + charlen =utf8_to_utf32 (source + i,6 , &u); +if (charlen <0 ) + goto unknown; + ret =utf8s_to_utf16s (source + i, charlen,UTF16_LITTLE_ENDIAN , wchar_to,6 );
@@ -589,8 +611,26 @@ cifs_local_to_utf16_bytes(const char *from, int len, {int charlen;int i; +int outlen; +unicode_t u_to;wchar_t wchar_to; +if (!codepage) { + outlen =0 ; +for (i =0 ; len && *from; i++, from += charlen, len -= charlen) { + charlen =utf8_to_utf32 (from, len, &u_to); +/* Failed conversion defaults to a question mark */ +if (charlen <1 ) { + charlen =1 ; + outlen +=2 ; + }else if (u_to <=0xFFFF ) + outlen +=2 ; +else + outlen +=4 ; + } +return outlen; + } +for (i =0 ; len && *from; i++, from += charlen, len -= charlen) { charlen = codepage->char2uni (from, len, &wchar_to);/* Failed conversion defaults to a question mark */
fs/cifs/cifs_unicode.h +1 -1
@@ -106,7 +106,7 @@ extern __le16 *cifs_strndup_to_utf16(const char *src, const int maxlen, int remap);#endif -wchar_t cifs_toupper (wchar_t in); +unicode_t cifs_toupper (unicode_t in);/* * UniStrcat: Concatenate the second string to the first
fs/cifs/cifsfs.c +2 -0
@@ -591,6 +591,8 @@ cifs_show_options(struct seq_file *s, struct dentry *root) cifs_sb->ctx->dir_mode);if (cifs_sb->ctx->iocharset)seq_printf (s,",iocharset=%s" , cifs_sb->ctx->iocharset); +else +seq_puts (s,",iocharset=utf8" );if (tcon->seal)seq_puts (s,",seal" );else if (tcon->ses->server->ignore_signature)
fs/cifs/connect.c +6 -2
@@ -2676,7 +2676,11 @@ compare_mount_options(struct super_block *sb, struct cifs_mnt_data *mnt_data) old->ctx->dir_mode != new->ctx->dir_mode)return 0 ; -if (strcmp (old->local_nls->charset, new->local_nls->charset)) +if (old->local_nls && !new->local_nls) +return 0 ; +if (!old->local_nls && new->local_nls) +return 0 ; +if (old->local_nls && new->local_nls &&strcmp (old->local_nls->charset, new->local_nls->charset))return 0 ;if (old->ctx->acregmax != new->ctx->acregmax)
@@ -3162,7 +3166,7 @@ int cifs_setup_cifs_sb(struct cifs_sb_info *cifs_sb)if (ctx->iocharset ==NULL ) {/* load_nls_default cannot return null */ cifs_sb->local_nls =load_nls_default (); - }else { + }else if (strcmp (ctx->iocharset,"utf8" ) !=0 ) { cifs_sb->local_nls =load_nls (ctx->iocharset);if (cifs_sb->local_nls ==NULL ) { cifs_dbg(VFS, "CIFS mount error: iocharset %s not found\n",
fs/cifs/dir.c +22 -6
@@ -804,16 +804,22 @@ static int cifs_ci_hash(const struct dentry *dentry, struct qstr *q) {struct nls_table *codepage =CIFS_SB (dentry->d_sb)->local_nls;unsigned long hash; +unicode_t u;wchar_t c;int i, charlen; hash =init_name_hash (dentry);for (i =0 ; i < q->len; i += charlen) { - charlen = codepage->char2uni (&q->name[i], q->len - i, &c); +if (codepage) { + charlen = codepage->char2uni (&q->name[i], q->len - i, &c); +if (likely (charlen >0 )) + u = c; + }else + charlen =utf8_to_utf32 (&q->name[i], q->len - i, &u);/* error out if we can't convert the character */ if (unlikely (charlen <0 ))return charlen; - hash =partial_name_hash (cifs_toupper (c), hash); + hash =partial_name_hash (cifs_toupper (u), hash); } q->hash =end_name_hash (hash);
@@ -824,6 +830,7 @@ static int cifs_ci_compare(const struct dentry *dentry, unsigned int len, const char *str, const struct qstr *name) { struct nls_table *codepage = CIFS_SB(dentry->d_sb)->local_nls; +unicode_t u1, u2;wchar_t c1, c2;int i, l1, l2;
@@ -837,9 +844,18 @@ static int cifs_ci_compare(const struct dentry *dentry,return 1 ;for (i =0 ; i < len; i += l1) { -/* Convert characters in both strings to UTF-16. */ - l1 = codepage->char2uni (&str[i], len - i, &c1); - l2 = codepage->char2uni (&name->name[i], name->len - i, &c2); +/* Convert characters in both strings to UTF-32. */ +if (codepage) { + l1 = codepage->char2uni (&str[i], len - i, &c1); + l2 = codepage->char2uni (&name->name[i], name->len - i, &c2); +if (likely (l1 >0 )) + u1 = c1; +if (likely (l2 >0 )) + u2 = c2; + }else { + l1 =utf8_to_utf32 (&str[i], len - i, &u1); + l2 =utf8_to_utf32 (&name->name[i], name->len - i, &u2); + }/* * If we can't convert either character, just declare it to
@@ -860,7 +876,7 @@ static int cifs_ci_compare(const struct dentry *dentry,return 1 ;/* Now compare uppercase versions of these characters */ -if (cifs_toupper (c1) !=cifs_toupper (c2)) +if (cifs_toupper (u1) !=cifs_toupper (u2))return 1 ; }
fs/cifs/winucase.c +9 -5
@@ -17,7 +17,7 @@#include <linux/nls.h> -wchar_t cifs_toupper (wchar_t in);/* quiet sparse */ +unicode_t cifs_toupper (unicode_t in);/* quiet sparse */ static const wchar_t t2_00[256 ] = {0x0000 ,0x0000 ,0x0000 ,0x0000 ,0x0000 ,0x0000 ,0x0000 ,0x0000 ,
@@ -615,20 +615,24 @@ static const wchar_t *const toplevel[256] = { };/** -* cifs_toupper - convert a wchar_t from lower to uppercase +* cifs_toupper - convert a unicode_t from lower to uppercase * @in: character to convert from lower to uppercase * -* This function consults the static tables above to convert a wchar_t from +* This function consults the static tables above to convert a unicode_t from * lower to uppercase. In the event that there is no mapping, the original * "in" character is returned. */ -wchar_t -cifs_toupper (wchar_t in) +unicode_t +cifs_toupper (unicode_t in) {unsigned char idx;const wchar_t *tbl;wchar_t out; +/* cifs_toupper table has only defines for plane-0 */ +if (in >0xffff ) +return in; +/* grab upper byte */ idx = (in &0xff00 ) >>8 ;
Signature · 1 line
2.20.1
RFC PATCH v2 18/18 nls: Drop broken nls_utf8 module
In reply to Pali RohárNLS table for utf8 is broken and cannot be fixed.
Now that all filesystems are using utf8s_to_utf16s()/utf16s_to_utf8s() functions for converting between UTF-8 and UTF-16, and functions utf8_to_utf32()/utf32_to_utf8() for converting between UTF-8 and Unicode code points, there is no need to have this broken utf8 NLS module in kernel tree anymore.
There is no user of this utf8 NLS module, so completely drop it,
--- fs/nls/Kconfig | 9 ------- fs/nls/Makefile | 1 - fs/nls/nls_utf8.c | 67 ----------------------------------------------- 3 files changed, 77 deletions(-) delete mode 100644 fs/nls/nls_utf8.c
fs/nls/Kconfig +0 -9
@@ -608,13 +608,4 @@ config NLS_MAC_TURKISH If unsure, say Y. -config NLS_UTF8 - tristate "NLS UTF-8" - help - If you want to display filenames with native language characters - from the Microsoft FAT file system family or from JOLIET CD-ROMs - correctly on the screen, you need to include the appropriate - input/output character sets. Say Y here for the UTF-8 encoding of - the Unicode/ISO9646 universal character set. - endif # NLS
fs/nls/Makefile +0 -1
@@ -42,7 +42,6 @@ obj-$(CONFIG_NLS_ISO8859_14) += nls_iso8859-14.o obj-$(CONFIG_NLS_ISO8859_15) += nls_iso8859-15.o obj-$(CONFIG_NLS_KOI8_R) += nls_koi8-r.o obj-$(CONFIG_NLS_KOI8_U) += nls_koi8-u.o nls_koi8-ru.o -obj-$(CONFIG_NLS_UTF8) += nls_utf8.o obj-$(CONFIG_NLS_MAC_CELTIC) += mac-celtic.o obj-$(CONFIG_NLS_MAC_CENTEURO) += mac-centeuro.o obj-$(CONFIG_NLS_MAC_CROATIAN) += mac-croatian.o
fs/nls/nls_utf8.c deleted +0 -67
67 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [RFC PATCH v2 03/18] ntfs: Undeprecate iocharset= mount option
In reply to Pali RohárQuoted 4 lines · 26.12.2022 klo 16.22 Pali Rohár ([email protected]) wrote:
Other fs drivers are using iocharset= mount option for specifying charset. So mark iocharset= mount option as preferred and deprecate nls= mount option.
snip.
Quoted 1 line
fs/ntfs/super.c +0
-0
snip.
fs/ntfs/super.c static bool parse_options(ntfs_volume *vol, char *opt) · line 219
Quoted 12 lines
fs/ntfs/super.c +4
-4@@ -218,10 +213,10 @@ static bool parse_options(ntfs_volume *vol, char *opt) }else if (!strcmp (p,"utf8" )) {bool val = false;ntfs_warning (vol->sb,"Option utf8 is no longer " -"supported, using option nls=utf8. Please " -"use option nls=utf8 in the future and " -"make sure utf8 is compiled either as a " -"module or into the kernel." ); +"supported, using option iocharset=utf8. " +"Please use option iocharset=utf8 in the " +"future and make sure utf8 is compiled " +"either as a module or into the kernel." );
We do not have to make sure utf8 is compiled anymore as it "always is" right?
Quoted 3 lines
fs/ntfs/super.c +0
-0if (!v || !*v) val = true;else if (!simple_getbool (v, &val))
Re: [RFC PATCH v2 03/18] ntfs: Undeprecate iocharset= mount option
Quoted 12 lines · On Sunday 01 January 2023 21:02:46 Kari Argillander wrote:
Quoted 4 lines · 26.12.2022 klo 16.22 Pali Rohár ([email protected]) wrote:
Other fs drivers are using iocharset= mount option for specifying charset. So mark iocharset= mount option as preferred and deprecate nls= mount option.
snip.
Quoted 1 line
fs/ntfs/super.c +0
-0snip.
Quoted 14 lines
fs/ntfs/super.c static bool parse_options(ntfs_volume *vol, char *opt) · line 219
Quoted 12 lines
fs/ntfs/super.c +4
-4@@ -218,10 +213,10 @@ static bool parse_options(ntfs_volume *vol, char *opt) }else if (!strcmp (p,"utf8" )) {bool val = false;ntfs_warning (vol->sb,"Option utf8 is no longer " -"supported, using option nls=utf8. Please " -"use option nls=utf8 in the future and " -"make sure utf8 is compiled either as a " -"module or into the kernel." ); +"supported, using option iocharset=utf8. " +"Please use option iocharset=utf8 in the " +"future and make sure utf8 is compiled " +"either as a module or into the kernel." );We do not have to make sure utf8 is compiled anymore as it "always is" right?
Yes, right, we have always utf8 support compiled-in. Second part of that warning message should be dropped.
Quoted 3 lines
Quoted 3 lines
fs/ntfs/super.c +0
-0if (!v || !*v) val = true;else if (!simple_getbool (v, &val))
Re: [RFC PATCH v2 03/18] ntfs: Undeprecate iocharset= mount option
Quoted 28 lines · On Sunday 01 January 2023 20:06:05 Pali Rohár wrote:
Quoted 12 lines · On Sunday 01 January 2023 21:02:46 Kari Argillander wrote:
Quoted 4 lines · 26.12.2022 klo 16.22 Pali Rohár ([email protected]) wrote:
Other fs drivers are using iocharset= mount option for specifying charset. So mark iocharset= mount option as preferred and deprecate nls= mount option.
snip.
Quoted 1 line
fs/ntfs/super.c +0
-0snip.
Quoted 14 lines
fs/ntfs/super.c static bool parse_options(ntfs_volume *vol, char *opt) · line 219
Quoted 12 lines
fs/ntfs/super.c +4
-4@@ -218,10 +213,10 @@ static bool parse_options(ntfs_volume *vol, char *opt) }else if (!strcmp (p,"utf8" )) {bool val = false;ntfs_warning (vol->sb,"Option utf8 is no longer " -"supported, using option nls=utf8. Please " -"use option nls=utf8 in the future and " -"make sure utf8 is compiled either as a " -"module or into the kernel." ); +"supported, using option iocharset=utf8. " +"Please use option iocharset=utf8 in the " +"future and make sure utf8 is compiled " +"either as a module or into the kernel." );We do not have to make sure utf8 is compiled anymore as it "always is" right?
Quoted 2 lines
Yes, right, we have always utf8 support compiled-in. Second part of that warning message should be dropped.
Ok, this is truth after patch 15/18. So info about compiled module should be dropped in patch 15/18, not in this one.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
fs/ntfs/super.c +0
-0if (!v || !*v) val = true;else if (!simple_getbool (v, &val))
OGAWA Hirofumi
maintainerRe: [RFC PATCH v2 01/18] fat: Fix iocharset=utf8 mount option
In reply to Pali RohárQuoted 6 lines · Pali Rohár
Currently iocharset=utf8 mount option is broken and error is printed to dmesg when it is used. To use UTF-8 as iocharset, it is required to use utf8=1 mount option.
Fix iocharset=utf8 mount option to use be equivalent to the utf8=1 mount option and remove printing error from dmesg.
[...]
Quoted 23 lines
Documentation/filesystems/vfat.rst +2
-11- - There is also an option of doing UTF-8 translations - with the utf8 option. - -.. note:: ``iocharset=utf8`` is not recommended. If unsure, you should consider - the utf8 option instead. + **utf8** is supported too and recommended to use. **utf8=<bool>** - UTF-8 is the filesystem safe version of Unicode that - is used by the console. It can be enabled or disabled - for the filesystem with this option. - If 'uni_xlate' gets set, UTF-8 gets disabled. - By default, FAT_DEFAULT_UTF8 setting is used. + Alias for ``iocharset=utf8`` mount option. **uni_xlate=<bool>** Translate unhandled Unicode characters to specialfs/fat/Kconfig +0
-0
fs/fat/Kconfig config FAT_DEFAULT_IOCHARSET · line 96
Quoted 7 lines
fs/fat/Kconfig +1
-3@@ -93,29 +93,12 @@ config FAT_DEFAULT_IOCHARSET like FAT to use. It should probably match the character set that most of your FAT filesystems use, and can be overridden with the "iocharset" mount option for FAT filesystems. - Note that "utf8" is not recommended for FAT filesystems. - If unsure, you shouldn't set "utf8" here - select the next option - instead if you would like to use UTF-8 encoded file names by default. + "utf8" is supported too and recommended to use.
This patch fixes the issue of utf-8 partially only. I think we can't still recommend only partially working one.
[...]
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat, · line 1319
Quoted 11 lines
fs/fat/Kconfig +1
-3- opts->utf8 = IS_ENABLED(CONFIG_FAT_DEFAULT_UTF8) && is_vfat; - if (!options) goto out;@@ -1318,10 +1316,14 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, | VFAT_SFN_CREATE_WIN95; break; case Opt_utf8_no: /* 0 or no or false */ - opts->utf8 = 0; + fat_reset_iocharset(opts);
This changes the behavior of "iocharset=iso8859-1,utf8=no" for example. Do we need this user visible change here?
Quoted 12 lines
fs/fat/Kconfig +5
-1break; case Opt_utf8_yes: /* empty or 1 or yes or true */ - opts->utf8 = 1; + fat_reset_iocharset(opts); + iocharset = kstrdup("utf8", GFP_KERNEL); + if (!iocharset) + return -ENOMEM; + opts->iocharset = iocharset; break; case Opt_uni_xl_no: /* 0 or no or false */ opts->unicode_xlate = 0;
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat, · line 1364
Quoted 10 lines
fs/fat/Kconfig +1
-6@@ -1359,18 +1361,11 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, } out: - /* UTF-8 doesn't provide FAT semantics */ - if (!strcmp(opts->iocharset, "utf8")) { - fat_msg(sb, KERN_WARNING, "utf8 is not a recommended IO charset" - " for FAT filesystems, filesystem will be " - "case sensitive!"); - } + opts->utf8 = !strcmp(opts->iocharset, "utf8") && is_vfat;
Still broken, so I think we still need the warning here (would be tweaked warning).
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat, · line 1372
Quoted 5 lines
fs/fat/Kconfig +0
-2/* If user doesn't specify allow_utime, it's initialized from dmask. */ if (opts->allow_utime == (unsigned short)-1) opts->allow_utime = ~opts->fs_dmask & (S_IWGRP | S_IWOTH); - if (opts->unicode_xlate) - opts->utf8 = 0;
unicode_xlate option is exclusive with utf8, need to adjust somewhere. (with this patch, unicode_xlate and utf8 will shows by show_options())
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat,
Quoted 5 lines
fs/fat/Kconfig +4
-0+ else if (utf8) + return fat_utf8_strnicmp(name->name, str, alen); + else + return nls_strnicmp(t, name->name, str, alen); }
Not strong opinion though, maybe we better to consolidate this to a (inline) function? (FWIW, it may be better to refactor to provide some filename functions to hide the detail of handling nls/utf8)
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Re: [RFC PATCH v2 01/18] fat: Fix iocharset=utf8 mount option
Quoted 36 lines · On Tuesday 10 January 2023 18:17:05 OGAWA Hirofumi wrote:
Quoted 6 lines · Pali Rohár
Currently iocharset=utf8 mount option is broken and error is printed to dmesg when it is used. To use UTF-8 as iocharset, it is required to use utf8=1 mount option.
Fix iocharset=utf8 mount option to use be equivalent to the utf8=1 mount option and remove printing error from dmesg.
[...]
Quoted 25 lines
Documentation/filesystems/vfat.rst +2
-11- - There is also an option of doing UTF-8 translations - with the utf8 option. - -.. note:: ``iocharset=utf8`` is not recommended. If unsure, you should consider - the utf8 option instead. + **utf8** is supported too and recommended to use. **utf8=<bool>** - UTF-8 is the filesystem safe version of Unicode that - is used by the console. It can be enabled or disabled - for the filesystem with this option. - If 'uni_xlate' gets set, UTF-8 gets disabled. - By default, FAT_DEFAULT_UTF8 setting is used. + Alias for ``iocharset=utf8`` mount option. **uni_xlate=<bool>** Translate unhandled Unicode characters to specialfs/fat/Kconfig +0
-0@@ -93,29 +93,12 @@ config FAT_DEFAULT_IOCHARSET like FAT to use. It should probably match the character set that most of your FAT filesystems use, and can be overridden
Quoted 8 lines
fs/fat/Kconfig config FAT_DEFAULT_IOCHARSET · line 96
Quoted 5 lines
fs/fat/Kconfig +1
-3with the "iocharset" mount option for FAT filesystems. - Note that "utf8" is not recommended for FAT filesystems. - If unsure, you shouldn't set "utf8" here - select the next option - instead if you would like to use UTF-8 encoded file names by default. + "utf8" is supported too and recommended to use.This patch fixes the issue of utf-8 partially only. I think we can't still recommend only partially working one.
With this patch FAT_DEFAULT_IOCHARSET=utf8 is same what was FAT_DEFAULT_UTF8=y without this patch. And option FAT_DEFAULT_UTF8 was recommended in description before "select the next option instead if you would like to use UTF-8 encoded file names by default."
Quoted 7 lines
[...]
Quoted 5 lines
fs/fat/Kconfig +0
-2- opts->utf8 = IS_ENABLED(CONFIG_FAT_DEFAULT_UTF8) && is_vfat; - if (!options) goto out;
Quoted 9 lines
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat, · line 1319
Quoted 6 lines
fs/fat/Kconfig +1
-1@@ -1318,10 +1316,14 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, | VFAT_SFN_CREATE_WIN95; break; case Opt_utf8_no: /* 0 or no or false */ - opts->utf8 = 0; + fat_reset_iocharset(opts);This changes the behavior of "iocharset=iso8859-1,utf8=no" for example. Do we need this user visible change here?
Ok, I agree, we do not want to change iocharset when "iocharset=iso8859-1,utf8=no" was specified. It should stay on iso8859-1.
Quoted 14 lines
Quoted 14 lines
fs/fat/Kconfig +5
-1break; case Opt_utf8_yes: /* empty or 1 or yes or true */ - opts->utf8 = 1; + fat_reset_iocharset(opts); + iocharset = kstrdup("utf8", GFP_KERNEL); + if (!iocharset) + return -ENOMEM; + opts->iocharset = iocharset; break; case Opt_uni_xl_no: /* 0 or no or false */ opts->unicode_xlate = 0;@@ -1359,18 +1361,11 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat, }
Quoted 11 lines
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat, · line 1364
Quoted 8 lines
fs/fat/Kconfig +1
-6out: - /* UTF-8 doesn't provide FAT semantics */ - if (!strcmp(opts->iocharset, "utf8")) { - fat_msg(sb, KERN_WARNING, "utf8 is not a recommended IO charset" - " for FAT filesystems, filesystem will be " - "case sensitive!"); - } + opts->utf8 = !strcmp(opts->iocharset, "utf8") && is_vfat;Still broken, so I think we still need the warning here (would be tweaked warning).
There was no warning before for utf8=1. And with this patch iocharset=utf8 should have same behavior as what was utf8=1 before this patch.
So if we should show some warning for utf8=1 then it is somehow not related to this patch and it should be done separately, possible also to the current codebase and before this patch.
Quoted 9 lines
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat, · line 1372
Quoted 5 lines
fs/fat/Kconfig +0
-2/* If user doesn't specify allow_utime, it's initialized from dmask. */ if (opts->allow_utime == (unsigned short)-1) opts->allow_utime = ~opts->fs_dmask & (S_IWGRP | S_IWOTH); - if (opts->unicode_xlate) - opts->utf8 = 0;
unicode_xlateoption is exclusive with utf8, need to adjust somewhere. (with this patch,unicode_xlateand utf8 will shows byshow_options())
Ok, seems that there is more work to handle unicode_xlate option correctly.
Quoted 9 lines
fs/fat/Kconfig static int parse_options(struct super_block *sb, char *options, int is_vfat,
Quoted 5 lines
fs/fat/Kconfig +4
-0+ else if (utf8) + return fat_utf8_strnicmp(name->name, str, alen); + else + return nls_strnicmp(t, name->name, str, alen); }Not strong opinion though, maybe we better to consolidate this to a (inline) function? (FWIW, it may be better to refactor to provide some filename functions to hide the detail of handling nls/utf8)
This looks like an another cleanup / improvement which can be done. I'm not sure if it is a good idea to put into this patch series (it is already big).
OGAWA Hirofumi
maintainerRe: [RFC PATCH v2 01/18] fat: Fix iocharset=utf8 mount option
Quoted 7 lines · Pali Rohár
Quoted 2 lines
This patch fixes the issue of utf-8 partially only. I think we can't still recommend only partially working one.
With this patch FAT_DEFAULT_IOCHARSET=utf8 is same what was FAT_DEFAULT_UTF8=y without this patch. And option
FAT_DEFAULT_UTF8was recommended in description before "select the next option instead if you would like to use UTF-8 encoded file names by default."
It is not recommending to use UTF-8 as default, right? I wanted to say no warning and recommend has big difference, and I can't recommend the incompatible behavior that creates the case sensitive filename.
Quoted 10 lines
Quoted 2 lines
Still broken, so I think we still need the warning here (would be tweaked warning).
There was no warning before for utf8=1. And with this patch iocharset=utf8 should have same behavior as what was utf8=1 before this patch.
So if we should show some warning for utf8=1 then it is somehow not related to this patch and it should be done separately, possible also to the current codebase and before this patch.
Sure, you are right.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>