Ext4 Encoding and Case-insensitive support
Subsystems (12)
v2 cover Ext4 Encoding and Case-insensitive support
[Resend, but also rebased on top of Ted's dev branch.]
Hi Ted,
This patchset implements encoding support as a superblock feature, valid for the entire disk, and Case-insensitive lookups support as a per-directory feature, configured as an inode flag. Alongside the addition of the casefolding patch, which was not part of v1, I fixed some bugs and addressed the concerns raised regarding invalid sequences and what normalization we use.
My approach to solve the two problems above is to make things more flexible. An encoding flags field in the superblock registers what kind of normalization is used by the filesystem. Right now, the only encoding supported for utf8 is NFKD, for instance, but if we want to support a different normalization function in the future, we can be backward compatible. Likewise, superblock flags also define the behavior when dealing with invalid sequences. The default behavior is to just treat invalid sequences as opaque byte sequences, falling back to the original behavior. Alternatively if the strict flag is enabled, the kernel will reject invalid sequences as soon as they are detected. All these flags, can only be set at mkfs time, using the encoding_flags parameter.
Each encoding has its default flags, when creating the filesystem in mkfs, regarding normalization/casefold functions and how to deal with opaque sequences.
The NLS patches implement generic casefold and normalization operation (defined as tolower() and identity(), respectively), allowing us to use any NLS charset in the kernel. We store the NLS charset as a magic number in the superblock, and for that only a few charsets are defined. If you want to force a different charset, the encoding and encoding_flags mount options are also provided.
This patchset also includes the NLS changes that I am proposing, including the entire utf8 normalization implementation. Differently from the previous version, I merged utf8 and utf8n, allowing the user to request a specific normalization (or no normalization at all) using the flags. As usual, I did not include the source ucd files because they would bounce in the list, but a completely functional implementation can be found at:
https://gitlab.collabora.com/krisman/linux.git -b ext4-ci-directory
I am also not supporting encoding with encrypted directories, given the cost of searching encrypted directories where the diggested name is not normalized. This means that we need to decrypt each filename beforehand, so I decided to simply skip this for now. If the user tries to mount with encoding a directory that has the encryption feature, we simply bail out saying it is not supported.
The patchset survives without failures the smoke tests of xfstests, with the obvious exception of generic/453. This test, which verifies that multiple files having equivalent names (which would match in utf8 normalization) are not the same file, doesn't really make sense with this patchset, since it basically verifies the fs is *not* encoding aware.
We also developed encoding and casefolding tests for xfstests, allowing us to quickly verify the implementation. I will be sumitting them upstream too, but for now they are available at:
https://gitlab.collabora.com/krisman/xfstests.git -b encoding
These tests validate the usage on inline dirs, on dx directories, when dealing with dcache and more. I am happy with the coverage we have now, but if you have specific concerns I can add more tests.
A modified version of e2fsprogs is necessary to run the tests, and to enable the feature. Support for mkfs, fsck, lsattr, chattr is available. I also hacked tune2fs to prevent it from setting the encryption flag when encoding is enabled. This tune2fs change is temporary until we are able to support these two features together.
https://gitlab.collabora.com/krisman/e2fsprogs.git -b encoding-feature
Gabriel Krisman Bertazi (21):
nls: Wrap uni2char/char2uni callers
nls: Wrap charset field access
nls: Wrap charset hooks in ops structure
nls: Split default charset from NLS core
nls: Split struct nls_charset from struct nls_table
nls: Add support for multiple versions of an encoding
nls: Implement NLS_STRICT_MODE flag
nls: Let charsets define the behavior of tolower/toupper
nls: Add new interface for string comparisons
nls: Add optional normalization and casefold hooks
nls: ascii: Support validation and normalization operations
nls: utf8: Move nls-utf8{,-core}.c
nls: utf8: Integrate utf8 normalization code with utf8 charset
nls: utf8: Introduce test module for normalized utf8 implementation
vfs: Handle case-exact lookup in d_add_ci
ext4: Include encoding information in the superblock
ext4: Add encoding mount options
ext4: Support encoding-aware file name lookups
ext4: Implement encoding-aware dcache hooks
ext4: Implement EXT4_CASEFOLD_FL flag
docs: ext4.rst: Document encoding and case-insensitive lookups
Olaf Weber (4):
nls: utf8n: Add unicode character database files
scripts: add trie generator for UTF-8
nls: utf8: Introduce code for UTF-8 normalization
nls: utf8n: reduce the size of utf8data[]
Documentation/filesystems/ext4/ext4.rst | 38 +
fs/befs/linuxvfs.c | 8 +-
fs/cifs/cifs_unicode.c | 15 +-
fs/cifs/cifsfs.c | 2 +-
fs/cifs/connect.c | 2 +-
fs/cifs/dir.c | 7 +-
fs/dcache.c | 33 +-
fs/ext4/dir.c | 59 +
fs/ext4/ext4.h | 33 +-
fs/ext4/hash.c | 38 +-
fs/ext4/ialloc.c | 2 +-
fs/ext4/inline.c | 2 +-
fs/ext4/inode.c | 4 +-
fs/ext4/ioctl.c | 18 +
fs/ext4/namei.c | 99 +-
fs/ext4/super.c | 170 ++
fs/fat/dir.c | 13 +-
fs/fat/inode.c | 6 +-
fs/fat/namei_vfat.c | 6 +-
fs/hfs/super.c | 6 +-
fs/hfs/trans.c | 9 +-
fs/hfsplus/options.c | 2 +-
fs/hfsplus/unicode.c | 6 +-
fs/isofs/inode.c | 5 +-
fs/isofs/joliet.c | 3 +-
fs/jfs/jfs_unicode.c | 9 +-
fs/jfs/super.c | 3 +-
fs/nls/Kconfig | 15 +
fs/nls/Makefile | 20 +
fs/nls/mac-celtic.c | 34 +-
fs/nls/mac-centeuro.c | 34 +-
fs/nls/mac-croatian.c | 34 +-
fs/nls/mac-cyrillic.c | 34 +-
fs/nls/mac-gaelic.c | 34 +-
fs/nls/mac-greek.c | 34 +-
fs/nls/mac-iceland.c | 34 +-
fs/nls/mac-inuit.c | 34 +-
fs/nls/mac-roman.c | 34 +-
fs/nls/mac-romanian.c | 34 +-
fs/nls/mac-turkish.c | 34 +-
fs/nls/nls_ascii.c | 84 +-
fs/nls/nls_core.c | 163 ++
fs/nls/nls_cp1250.c | 34 +-
fs/nls/nls_cp1251.c | 34 +-
fs/nls/nls_cp1255.c | 36 +-
fs/nls/nls_cp437.c | 34 +-
fs/nls/nls_cp737.c | 34 +-
fs/nls/nls_cp775.c | 34 +-
fs/nls/nls_cp850.c | 34 +-
fs/nls/nls_cp852.c | 34 +-
fs/nls/nls_cp855.c | 34 +-
fs/nls/nls_cp857.c | 34 +-
fs/nls/nls_cp860.c | 34 +-
fs/nls/nls_cp861.c | 34 +-
fs/nls/nls_cp862.c | 34 +-
fs/nls/nls_cp863.c | 34 +-
fs/nls/nls_cp864.c | 34 +-
fs/nls/nls_cp865.c | 34 +-
fs/nls/nls_cp866.c | 34 +-
fs/nls/nls_cp869.c | 34 +-
fs/nls/nls_cp874.c | 36 +-
fs/nls/nls_cp932.c | 36 +-
fs/nls/nls_cp936.c | 36 +-
fs/nls/nls_cp949.c | 36 +-
fs/nls/nls_cp950.c | 36 +-
fs/nls/{nls_base.c => nls_default.c} | 124 +-
fs/nls/nls_euc-jp.c | 29 +-
fs/nls/nls_iso8859-1.c | 34 +-
fs/nls/nls_iso8859-13.c | 34 +-
fs/nls/nls_iso8859-14.c | 34 +-
fs/nls/nls_iso8859-15.c | 34 +-
fs/nls/nls_iso8859-2.c | 34 +-
fs/nls/nls_iso8859-3.c | 34 +-
fs/nls/nls_iso8859-4.c | 34 +-
fs/nls/nls_iso8859-5.c | 34 +-
fs/nls/nls_iso8859-6.c | 34 +-
fs/nls/nls_iso8859-7.c | 34 +-
fs/nls/nls_iso8859-9.c | 34 +-
fs/nls/nls_koi8-r.c | 34 +-
fs/nls/nls_koi8-ru.c | 30 +-
fs/nls/nls_koi8-u.c | 34 +-
fs/nls/nls_utf8-core.c | 333 +++
fs/nls/nls_utf8-norm.c | 797 ++++++
fs/nls/nls_utf8-selftest.c | 307 ++
fs/nls/nls_utf8.c | 67 -
fs/nls/ucd/README | 33 +
fs/nls/utf8n.h | 117 +
fs/ntfs/inode.c | 2 +-
fs/ntfs/super.c | 6 +-
fs/ntfs/unistr.c | 13 +-
fs/udf/super.c | 3 +-
fs/udf/unicode.c | 4 +-
include/linux/fs.h | 2 +
include/linux/nls.h | 293 +-
scripts/Makefile | 1 +
scripts/mkutf8data.c | 3464 +++++++++++++++++++++++
96 files changed, 7482 insertions(+), 633 deletions(-)
create mode 100644 fs/nls/nls_core.c
rename fs/nls/{nls_base.c => nls_default.c} (89%)
create mode 100644 fs/nls/nls_utf8-core.c
create mode 100644 fs/nls/nls_utf8-norm.c
create mode 100644 fs/nls/nls_utf8-selftest.c
delete mode 100644 fs/nls/nls_utf8.c
create mode 100644 fs/nls/ucd/README
create mode 100644 fs/nls/utf8n.h
create mode 100644 scripts/mkutf8data.cSignature · 1 line
2.19.0
PATCH RESEND v2 1/25 nls: Wrap uni2char/char2uni callers
Just a cosmetic change at this point, this patch will simplify the following Coccinelle patches which will move the hooks into a dedicated structure, with the goal of splitting the nls_table structure to support versioning.
This was generated with the following coccinele script:
<smpl>
@@
expression A, B, C, D;
@@
(
- A->uni2char(B, C, D)
+ nls_uni2char(A, B, C, D)
|
- A->char2uni(B, C, D)
+ nls_char2uni(A, B, C, D)
)
</smpl>
--- fs/befs/linuxvfs.c | 4 ++-- fs/cifs/cifs_unicode.c | 9 +++++---- fs/cifs/dir.c | 7 ++++--- fs/fat/dir.c | 8 ++++---- fs/fat/namei_vfat.c | 6 +++--- fs/hfs/trans.c | 9 +++++---- fs/hfsplus/unicode.c | 6 +++--- fs/isofs/joliet.c | 3 ++- fs/jfs/jfs_unicode.c | 7 +++---- fs/nls/nls_euc-jp.c | 4 ++-- fs/nls/nls_koi8-ru.c | 6 +++--- fs/ntfs/unistr.c | 8 ++++---- include/linux/nls.h | 13 +++++++++++++ 13 files changed, 53 insertions(+), 37 deletions(-)
fs/befs/linuxvfs.c +2 -2
@@ -542,7 +542,7 @@ befs_utf2nls(struct super_block *sb, const char *in,/* convert from Unicode to nls */ if (uni >MAX_WCHAR_T ) goto conv_err; - unilen = nls->uni2char (uni, &result[o], in_len - o); + unilen =nls_uni2char (nls, uni, &result[o], in_len - o);if (unilen <0 ) goto conv_err; }
@@ -616,7 +616,7 @@ befs_nls2utf(struct super_block *sb, const char *in,for (i = o =0 ; i < in_len; i += unilen, o += utflen) {/* convert from nls to unicode */ - unilen = nls->char2uni (&in[i], in_len - i, &uni); + unilen =nls_char2uni (nls, &in[i], in_len - i, &uni);if (unilen <0 ) goto conv_err;
fs/cifs/cifs_unicode.c +5 -4
@@ -148,7 +148,7 @@ cifs_mapchar(char *target, const __u16 *from, const struct nls_table *cp,return len;/* if character not one of seven in special remap set */ - len = cp->uni2char (src_char, target,NLS_MAX_CHARSET_SIZE ); + len =nls_uni2char (cp, src_char, target,NLS_MAX_CHARSET_SIZE );if (len <=0 ) goto surrogate_pair;
@@ -292,7 +292,7 @@ cifs_strtoUTF16(__le16 *to, const char *from, int len, }for (i =0 ; len && *from; i++, from += charlen, len -= charlen) { - charlen = codepage->char2uni (from, len, &wchar_to); + charlen =nls_char2uni (codepage, from, len, &wchar_to);if (charlen <1 ) {cifs_dbg (VFS ,"strtoUTF16: char2uni of 0x%x returned %d\n" , *from, charlen);
@@ -518,7 +518,8 @@ cifsConvertToUTF16(__le16 *target, const char *source, int srclen,* as they use backslash as separator. */ if (dst_char ==0 ) { - charlen = cp->char2uni (source + i, srclen - i, &tmp); + charlen =nls_char2uni (cp, source + i, srclen - i, + &tmp); dst_char =cpu_to_le16 (tmp);/*
@@ -608,7 +609,7 @@ cifs_local_to_utf16_bytes(const char *from, int len,wchar_t wchar_to;for (i =0 ; len && *from; i++, from += charlen, len -= charlen) { - charlen = codepage->char2uni (from, len, &wchar_to); + charlen =nls_char2uni (codepage, from, len, &wchar_to);/* Failed conversion defaults to a question mark */ if (charlen <1 ) charlen =1 ;
fs/cifs/dir.c +4 -3
@@ -911,7 +911,7 @@ static int cifs_ci_hash(const struct dentry *dentry, struct qstr *q) hash =init_name_hash (dentry);for (i =0 ; i < q->len; i += charlen) { - charlen = codepage->char2uni (&q->name[i], q->len - i, &c); + charlen =nls_char2uni (codepage, &q->name[i], q->len - i, &c);/* error out if we can't convert the character */ if (unlikely (charlen <0 ))return charlen;
@@ -940,8 +940,9 @@ static int cifs_ci_compare(const struct dentry *dentry,for (i =0 ; i < len; i += l1) {/* Convert characters in both strings to UTF-16. */ - l1 = codepage->char2uni (&str[i], len - i, &c1); - l2 = codepage->char2uni (&name->name[i], name->len - i, &c2); + l1 =nls_char2uni (codepage, &str[i], len - i, &c1); + l2 =nls_char2uni (codepage, &name->name[i], name->len - i, + &c2);/* * If we can't convert either character, just declare it to
fs/fat/dir.c +4 -4
@@ -153,7 +153,7 @@ static int uni16_to_x8(struct super_block *sb, unsigned char *ascii,while (*ip && ((len -NLS_MAX_CHARSET_SIZE ) >0 )) { ec = *ip++; - charlen = nls->uni2char (ec, op,NLS_MAX_CHARSET_SIZE ); + charlen =nls_uni2char (nls, ec, op,NLS_MAX_CHARSET_SIZE );if (charlen >0 ) { op += charlen; len -= charlen;
@@ -195,7 +195,7 @@ fat_short2uni(struct nls_table *t, unsigned char *c, int clen, wchar_t *uni) {int charlen; - charlen = t->char2uni (c, clen, uni); + charlen =nls_char2uni (t, c, clen, uni);if (charlen <0 ) { *uni =0x003f ;/* a question mark */ charlen =1 ;
@@ -210,7 +210,7 @@ fat_short2lower_uni(struct nls_table *t, unsigned char *c,int charlen;wchar_t wc; - charlen = t->char2uni (c, clen, &wc); + charlen =nls_char2uni (t, c, clen, &wc);if (charlen <0 ) { *uni =0x003f ;/* a question mark */ charlen =1 ;
@@ -220,7 +220,7 @@ fat_short2lower_uni(struct nls_table *t, unsigned char *c,if (!nc) nc = *c; - charlen = t->char2uni (&nc,1 , uni); + charlen =nls_char2uni (t, &nc,1 , uni);if (charlen <0 ) { *uni =0x003f ;/* a question mark */ charlen =1 ;
fs/fat/namei_vfat.c +3 -3
@@ -289,7 +289,7 @@ static inline int to_shortname_char(struct nls_table *nls,return 1 ; } - len = nls->uni2char (*src, buf, buf_size); + len =nls_uni2char (nls, *src, buf, buf_size);if (len <=0 ) { info->valid =0 ; buf[0 ] ='_' ;
@@ -544,8 +544,8 @@ xlate_to_uni(const unsigned char *name, int len, unsigned char *outname, ip +=5 ; i +=5 ; }else { - charlen = nls->char2uni (ip, len - i, - (wchar_t *)op); + charlen =nls_char2uni (nls, ip, len - i, + (wchar_t *)op);if (charlen <0 )return -EINVAL ; ip += charlen;
fs/hfs/trans.c +5 -4
@@ -49,7 +49,8 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in)while (srclen >0 ) {if (nls_disk) { - size = nls_disk->char2uni (src, srclen, &ch); + size =nls_char2uni (nls_disk, src, srclen, + &ch);if (size <=0 ) { ch ='?' ; size =1 ;
@@ -62,7 +63,7 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in) }if (ch =='/' ) ch =':' ; - size = nls_io->uni2char (ch, dst, dstlen); + size =nls_uni2char (nls_io, ch, dst, dstlen);if (size <0 ) {if (size == -ENAMETOOLONG ) goto out;
@@ -110,7 +111,7 @@ void hfs_asc2mac(struct super_block *sb, struct hfs_name *out, const struct qstrwchar_t ch;while (srclen >0 ) { - size = nls_io->char2uni (src, srclen, &ch); + size =nls_char2uni (nls_io, src, srclen, &ch);if (size <0 ) { ch ='?' ; size =1 ;
@@ -120,7 +121,7 @@ void hfs_asc2mac(struct super_block *sb, struct hfs_name *out, const struct qstrif (ch ==':' ) ch ='/' ;if (nls_disk) { - size = nls_disk->uni2char (ch, dst, dstlen); + size =nls_uni2char (nls_disk, ch, dst, dstlen);if (size <0 ) {if (size == -ENAMETOOLONG ) goto out;
fs/hfsplus/unicode.c +3 -3
@@ -190,7 +190,7 @@ int hfsplus_uni2asc(struct super_block *sb, c0 =':' ;break ; } - res = nls->uni2char (c0, op, len); + res =nls_uni2char (nls, c0, op, len);if (res <0 ) {if (res == -ENAMETOOLONG ) goto out;
@@ -233,7 +233,7 @@ int hfsplus_uni2asc(struct super_block *sb, cc = c0; } done: - res = nls->uni2char (cc, op, len); + res =nls_uni2char (nls, cc, op, len);if (res <0 ) {if (res == -ENAMETOOLONG ) goto out;
@@ -256,7 +256,7 @@ int hfsplus_uni2asc(struct super_block *sb,static inline int asc2unichar (struct super_block *sb,const char *astr,int len,wchar_t *uc) { -int size =HFSPLUS_SB (sb)->nls->char2uni (astr, len, uc); +int size =nls_char2uni (HFSPLUS_SB (sb)->nls, astr, len, uc);if (size <=0 ) { *uc ='?' ; size =1 ;
fs/isofs/joliet.c +2 -1
@@ -25,7 +25,8 @@ uni16_to_x8(unsigned char *ascii, __be16 *uni, int len, struct nls_table *nls)while ((ch =get_unaligned (ip)) && len) {int llen; - llen = nls->uni2char (be16_to_cpu (ch), op,NLS_MAX_CHARSET_SIZE ); + llen =nls_uni2char (nls,be16_to_cpu (ch), op, +NLS_MAX_CHARSET_SIZE );if (llen >0 ) op += llen;else
fs/jfs/jfs_unicode.c +3 -4
@@ -41,9 +41,8 @@ int jfs_strfromUCS_le(char *to, const __le16 * from,for (i =0 ; (i < len) && from[i]; i++) {int charlen; charlen = - codepage->uni2char (le16_to_cpu (from[i]), - &to[outlen], -NLS_MAX_CHARSET_SIZE ); +nls_uni2char (codepage,le16_to_cpu (from[i]), + &to[outlen],NLS_MAX_CHARSET_SIZE );if (charlen >0 ) outlen += charlen;else
@@ -88,7 +87,7 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len,if (codepage) {for (i =0 ; len && *from; i++, from += charlen, len -= charlen) { - charlen = codepage->char2uni (from, len, &to[i]); + charlen =nls_char2uni (codepage, from, len, &to[i]);if (charlen <1 ) {jfs_err ("jfs_strtoUCS: char2uni returned %d." , charlen);
fs/nls/nls_euc-jp.c +2 -2
@@ -413,7 +413,7 @@ static int uni2char(const wchar_t uni,if (!p_nls)return -EINVAL ; -if ((n = p_nls->uni2char (uni, out, boundlen)) <0 ) +if ((n =nls_uni2char (p_nls, uni, out, boundlen)) <0 )return n;/* translate SJIS into EUC-JP */
@@ -543,7 +543,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen, sjis_temp[1 ] =0x00 ; } -if ( (n = p_nls->char2uni (sjis_temp,sizeof (sjis_temp), uni)) <0 ) +if ( (n =nls_char2uni (p_nls, sjis_temp,sizeof (sjis_temp), uni)) <0 )return n;return euc_offset;
fs/nls/nls_koi8-ru.c +3 -3
@@ -28,12 +28,12 @@ static int uni2char(const wchar_t uni,else if (uni ==0x255d || uni ==0x256c )return 0 ;else -return p_nls->uni2char (uni, out, boundlen); +return nls_uni2char (p_nls, uni, out, boundlen);return 1 ; }else /* fast path */ -return p_nls->uni2char (uni, out, boundlen); +return nls_uni2char (p_nls, uni, out, boundlen); } static int char2uni(const unsigned char *rawstring, int boundlen,
@@ -47,7 +47,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return 1 ; } - n = p_nls->char2uni (rawstring, boundlen, uni); + n =nls_char2uni (p_nls, rawstring, boundlen, uni);return n; }
fs/ntfs/unistr.c +4 -4
@@ -269,8 +269,8 @@ int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins, ucs =kmem_cache_alloc (ntfs_name_cache,GFP_NOFS );if (likely (ucs)) {for (i = o =0 ; i < ins_len; i += wc_len) { - wc_len = nls->char2uni (ins + i, ins_len - i, - &wc); + wc_len =nls_char2uni (nls, ins + i, + ins_len - i, &wc);if (likely (wc_len >=0 && o <NTFS_MAX_NAME_LEN )) {if (likely (wc)) {
@@ -355,8 +355,8 @@ int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins, goto mem_err_out; }for (i = o =0 ; i < ins_len; i++) { -retry: wc = nls->uni2char (le16_to_cpu (ins[i]), ns + o, - ns_len - o); +retry: wc =nls_uni2char (nls,le16_to_cpu (ins[i]), + ns + o, ns_len - o);if (wc >0 ) { o += wc;continue ;
include/linux/nls.h +13 -0
@@ -59,6 +59,19 @@ extern int utf8s_to_utf16s(const u8 *s, int len,extern int utf16s_to_utf8s (const wchar_t *pwcs,int len,enum utf16_endian endian,u8 *s,int maxlen); +static inline int nls_uni2char (const struct nls_table *table,wchar_t uni, +unsigned char *out,int boundlen) +{ +return table->uni2char (uni, out, boundlen); +} + +static inline int nls_char2uni (const struct nls_table *table, +const unsigned char *rawstring, +int boundlen,wchar_t *uni) +{ +return table->char2uni (rawstring, boundlen, uni); +} +static inline unsigned char nls_tolower (struct nls_table *t,unsigned char c) {unsigned char nc = t->charset2lower[c];
Signature · 1 line
2.19.0
PATCH RESEND v2 2/25 nls: Wrap charset field access
In reply to Gabriel Krisman BertaziThe goal is to simplify the following patches that split nls_table. No behavior changes intended.
<smpl>
@@
struct nls_table *c;
@@
- c->charset
+ nls_charset_name(c)
</smpl>
--- fs/befs/linuxvfs.c | 4 ++-- fs/cifs/cifs_unicode.c | 6 +++--- fs/cifs/cifsfs.c | 2 +- fs/cifs/connect.c | 2 +- fs/fat/inode.c | 6 ++++-- fs/hfs/super.c | 6 ++++-- fs/hfsplus/options.c | 2 +- fs/isofs/inode.c | 5 +++-- fs/jfs/jfs_unicode.c | 2 +- fs/jfs/super.c | 3 ++- fs/nls/nls_base.c | 2 +- fs/ntfs/inode.c | 2 +- fs/ntfs/super.c | 6 +++--- fs/ntfs/unistr.c | 5 +++-- fs/udf/super.c | 3 ++- include/linux/nls.h | 5 +++++ 16 files changed, 37 insertions(+), 24 deletions(-)
fs/befs/linuxvfs.c +2 -2
@@ -555,7 +555,7 @@ befs_utf2nls(struct super_block *sb, const char *in, conv_err:befs_error (sb,"Name using character set %s contains a character that " -"cannot be converted to unicode." , nls->charset); +"cannot be converted to unicode." ,nls_charset_name (nls));befs_debug (sb,"<--- %s" , __func__);kfree (result);return -EILSEQ ;
@@ -635,7 +635,7 @@ befs_nls2utf(struct super_block *sb, const char *in, conv_err:befs_error (sb,"Name using character set %s contains a character that " -"cannot be converted to unicode." , nls->charset); +"cannot be converted to unicode." ,nls_charset_name (nls));befs_debug (sb,"<--- %s" , __func__);kfree (result);return -EILSEQ ;
fs/cifs/cifs_unicode.c +3 -3
@@ -156,7 +156,7 @@ cifs_mapchar(char *target, const __u16 *from, const struct nls_table *cp, surrogate_pair:/* convert SURROGATE_PAIR and IVS */ -if (strcmp (cp->charset,"utf8" )) +if (strcmp (nls_charset_name (cp),"utf8" )) goto unknown; len =utf16s_to_utf8s (from,3 ,UTF16_LITTLE_ENDIAN , target,6 );if (len <=0 )
@@ -271,7 +271,7 @@ cifs_strtoUTF16(__le16 *to, const char *from, int len,wchar_t wchar_to;/* needed to quiet sparse */ /* special case for utf8 to handle no plane0 chars */ -if (!strcmp (codepage->charset,"utf8" )) { +if (!strcmp (nls_charset_name (codepage),"utf8" )) {/* * convert utf8 -> utf16, we assume we have enough space * as caller should have assumed conversion does not overflow
@@ -530,7 +530,7 @@ cifsConvertToUTF16(__le16 *target, const char *source, int srclen, goto ctoUTF16;/* convert SURROGATE_PAIR */ -if (strcmp (cp->charset,"utf8" ) || !wchar_to) +if (strcmp (nls_charset_name (cp),"utf8" ) || !wchar_to) goto unknown;if (*(source + i) &0x80 ) { charlen =utf8_to_utf32 (source + i,6 , &u);
fs/cifs/cifsfs.c +1 -1
@@ -401,7 +401,7 @@ cifs_show_nls(struct seq_file *s, struct nls_table *cur)/* Display iocharset= option if it's not default charset */ def =load_nls_default ();if (def != cur) -seq_printf (s,",iocharset=%s" , cur->charset); +seq_printf (s,",iocharset=%s" ,nls_charset_name (cur));unload_nls (def); }
fs/cifs/connect.c +1 -1
@@ -3160,7 +3160,7 @@ compare_mount_options(struct super_block *sb, struct cifs_mnt_data *mnt_data) old->mnt_dir_mode != new->mnt_dir_mode)return 0 ; -if (strcmp (old->local_nls->charset, new->local_nls->charset)) +if (strcmp (nls_charset_name (old->local_nls),nls_charset_name (new->local_nls)))return 0 ;if (old->actimeo != new->actimeo)
fs/fat/inode.c +4 -2
@@ -948,10 +948,12 @@ static int fat_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",allow_utime=%04o" , opts->allow_utime);if (sbi->nls_disk)/* strip "cp" prefix from displayed option */ -seq_printf (m,",codepage=%s" , &sbi->nls_disk->charset[2 ]); +seq_printf (m,",codepage=%s" , + &nls_charset_name (sbi->nls_disk)[2 ]);if (isvfat) {if (sbi->nls_io) -seq_printf (m,",iocharset=%s" , sbi->nls_io->charset); +seq_printf (m,",iocharset=%s" , +nls_charset_name (sbi->nls_io));switch (opts->shortname) {case VFAT_SFN_DISPLAY_WIN95 |VFAT_SFN_CREATE_WIN95 :
fs/hfs/super.c +4 -2
@@ -151,9 +151,11 @@ static int hfs_show_options(struct seq_file *seq, struct dentry *root)if (sbi->session >=0 )seq_printf (seq,",session=%u" , sbi->session);if (sbi->nls_disk) -seq_printf (seq,",codepage=%s" , sbi->nls_disk->charset); +seq_printf (seq,",codepage=%s" , +nls_charset_name (sbi->nls_disk));if (sbi->nls_io) -seq_printf (seq,",iocharset=%s" , sbi->nls_io->charset); +seq_printf (seq,",iocharset=%s" , +nls_charset_name (sbi->nls_io));if (sbi->s_quiet)seq_printf (seq,",quiet" );return 0 ;
fs/hfsplus/options.c +1 -1
@@ -230,7 +230,7 @@ int hfsplus_show_options(struct seq_file *seq, struct dentry *root)if (sbi->session >=0 )seq_printf (seq,",session=%u" , sbi->session);if (sbi->nls) -seq_printf (seq,",nls=%s" , sbi->nls->charset); +seq_printf (seq,",nls=%s" ,nls_charset_name (sbi->nls));if (test_bit (HFSPLUS_SB_NODECOMPOSE , &sbi->flags))seq_puts (seq,",nodecompose" );if (test_bit (HFSPLUS_SB_NOBARRIER , &sbi->flags))
fs/isofs/inode.c +3 -2
@@ -519,8 +519,9 @@ static int isofs_show_options(struct seq_file *m, struct dentry *root)#ifdef CONFIG_JOLIET if (sbi->s_nls_iocharset && -strcmp (sbi->s_nls_iocharset->charset,CONFIG_NLS_DEFAULT ) !=0 ) -seq_printf (m,",iocharset=%s" , sbi->s_nls_iocharset->charset); +strcmp (nls_charset_name (sbi->s_nls_iocharset),CONFIG_NLS_DEFAULT ) !=0 ) +seq_printf (m,",iocharset=%s" , +nls_charset_name (sbi->s_nls_iocharset));#endif return 0 ; }
fs/jfs/jfs_unicode.c +1 -1
@@ -92,7 +92,7 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len,jfs_err ("jfs_strtoUCS: char2uni returned %d." , charlen);jfs_err ("charset = %s, char = 0x%x" , - codepage->charset, *from); +nls_charset_name (codepage), *from);return charlen; } }
fs/jfs/super.c +2 -1
@@ -736,7 +736,8 @@ static int jfs_show_options(struct seq_file *seq, struct dentry *root)if (sbi->flag &JFS_DISCARD )seq_printf (seq,",discard=%u" , sbi->minblks_trim);if (sbi->nls_tab) -seq_printf (seq,",iocharset=%s" , sbi->nls_tab->charset); +seq_printf (seq,",iocharset=%s" , +nls_charset_name (sbi->nls_tab));if (sbi->flag &JFS_ERR_CONTINUE )seq_printf (seq,",errors=continue" );if (sbi->flag &JFS_ERR_PANIC )
fs/nls/nls_base.c +1 -1
@@ -277,7 +277,7 @@ static struct nls_table *find_nls(char *charset)struct nls_table *nls;spin_lock (&nls_lock);for (nls = tables; nls; nls = nls->next) { -if (!strcmp (nls->charset, charset)) +if (!strcmp (nls_charset_name (nls), charset))break ;if (nls->alias && !strcmp (nls->alias, charset))break ;
fs/ntfs/inode.c +1 -1
@@ -2313,7 +2313,7 @@ int ntfs_show_options(struct seq_file *sf, struct dentry *root)seq_printf (sf,",fmask=0%o" , vol->fmask);seq_printf (sf,",dmask=0%o" , vol->dmask); } -seq_printf (sf,",nls=%s" , vol->nls_map->charset); +seq_printf (sf,",nls=%s" ,nls_charset_name (vol->nls_map));if (NVolCaseSensitive (vol))seq_printf (sf,",case_sensitive" );if (NVolShowSystemFiles (vol))
fs/ntfs/super.c +3 -3
@@ -224,7 +224,7 @@ static bool parse_options(ntfs_volume *vol, char *opt) }ntfs_error (vol->sb,"NLS character set %s not " "found. Using previous one %s." , - v, old_nls->charset); + v,nls_charset_name (old_nls)); nls_map = old_nls; }else /* nls_map */ {unload_nls (old_nls);
@@ -274,7 +274,7 @@ static bool parse_options(ntfs_volume *vol, char *opt) "on remount.");return false; }/* else (!vol->nls_map) */ -ntfs_debug ("Using NLS character set %s." , nls_map->charset); +ntfs_debug ("Using NLS character set %s." ,nls_charset_name (nls_map)); vol->nls_map = nls_map; }else /* (!nls_map) */ {if (!vol->nls_map) {
@@ -285,7 +285,7 @@ static bool parse_options(ntfs_volume *vol, char *opt)return false; }ntfs_debug ("Using default NLS character set (%s)." , - vol->nls_map->charset); +nls_charset_name (vol->nls_map)); } }if (mft_zone_multiplier !=-1 ) {
fs/ntfs/unistr.c +3 -2
@@ -297,7 +297,7 @@ int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,if (wc_len <0 ) {ntfs_error (vol->sb,"Name using character set %s contains " "characters that cannot be converted to " -"Unicode." , nls->charset); +"Unicode." ,nls_charset_name (nls)); i = -EILSEQ ; }else /* if (o >= NTFS_MAX_NAME_LEN) */ { ntfs_error(vol->sb, "Name is too long (maximum length for a "
@@ -386,7 +386,8 @@ retry: wc = nls_uni2char(nls, le16_to_cpu(ins[i]), conversion_err:ntfs_error (vol->sb,"Unicode name contains characters that cannot be " "converted to character set %s. You might want to " -"try to use the mount option nls=utf8." , nls->charset); +"try to use the mount option nls=utf8." , +nls_charset_name (nls));if (ns != *outs)kfree (ns);if (wc != -ENAMETOOLONG )
fs/udf/super.c +2 -1
@@ -365,7 +365,8 @@ static int udf_show_options(struct seq_file *seq, struct dentry *root)if (UDF_QUERY_FLAG (sb,UDF_FLAG_UTF8 ))seq_puts (seq,",utf8" );if (UDF_QUERY_FLAG (sb,UDF_FLAG_NLS_MAP ) && sbi->s_nls_map) -seq_printf (seq,",iocharset=%s" , sbi->s_nls_map->charset); +seq_printf (seq,",iocharset=%s" , +nls_charset_name (sbi->s_nls_map));return 0 ; }
include/linux/nls.h +5 -0
@@ -72,6 +72,11 @@ static inline int nls_char2uni(const struct nls_table *table,return table->char2uni (rawstring, boundlen, uni); } +static inline const char *nls_charset_name (const struct nls_table *table) +{ +return table->charset; +} +static inline unsigned char nls_tolower (struct nls_table *t,unsigned char c) {unsigned char nc = t->charset2lower[c];
Signature · 1 line
2.19.0
PATCH RESEND v2 3/25 nls: Wrap charset hooks in ops structure
In reply to Gabriel Krisman BertaziThis is done in preparation to splitting the nls_table structure, in order to support multiple versions of the same encoding. By placing all the operations together, we can allow multiple operations for the same encoding, depending on the version. For now, there is no behavior change intended, but this simplify the following patches.
With the exception of the declaration of the structure, this patch was generated by the following Coccinelle script:
<smpl>
@nlstable@
identifier p;
expression uni2char_fn;
expression char2uni_fn;
@@
static struct nls_table p = {
- .char2uni = char2uni_fn, - .uni2char = uni2char_fn, + .ops = &charset_ops,
};
@createops@
identifier nlstable.p;
expression nlstable.uni2char_fn;
expression nlstable.char2uni_fn;
@@
+static const struct nls_ops charset_ops = {
+ .uni2char = uni2char_fn,
+ .char2uni = char2uni_fn,
+};
+
static struct nls_table p = {};
@@
struct nls_table *c;
@@
(
- c->uni2char
+ c->ops->uni2char
|
- c->char2uni
+ c->ops->char2uni
)
</smpl>
--- fs/nls/mac-celtic.c | 8 ++++++-- fs/nls/mac-centeuro.c | 8 ++++++-- fs/nls/mac-croatian.c | 8 ++++++-- fs/nls/mac-cyrillic.c | 8 ++++++-- fs/nls/mac-gaelic.c | 8 ++++++-- fs/nls/mac-greek.c | 8 ++++++-- fs/nls/mac-iceland.c | 8 ++++++-- fs/nls/mac-inuit.c | 8 ++++++-- fs/nls/mac-roman.c | 8 ++++++-- fs/nls/mac-romanian.c | 8 ++++++-- fs/nls/mac-turkish.c | 8 ++++++-- fs/nls/nls_ascii.c | 8 ++++++-- fs/nls/nls_base.c | 8 ++++++-- fs/nls/nls_cp1250.c | 8 ++++++-- fs/nls/nls_cp1251.c | 8 ++++++-- fs/nls/nls_cp1255.c | 8 ++++++-- fs/nls/nls_cp437.c | 8 ++++++-- fs/nls/nls_cp737.c | 8 ++++++-- fs/nls/nls_cp775.c | 8 ++++++-- fs/nls/nls_cp850.c | 8 ++++++-- fs/nls/nls_cp852.c | 8 ++++++-- fs/nls/nls_cp855.c | 8 ++++++-- fs/nls/nls_cp857.c | 8 ++++++-- fs/nls/nls_cp860.c | 8 ++++++-- fs/nls/nls_cp861.c | 8 ++++++-- fs/nls/nls_cp862.c | 8 ++++++-- fs/nls/nls_cp863.c | 8 ++++++-- fs/nls/nls_cp864.c | 8 ++++++-- fs/nls/nls_cp865.c | 8 ++++++-- fs/nls/nls_cp866.c | 8 ++++++-- fs/nls/nls_cp869.c | 8 ++++++-- fs/nls/nls_cp874.c | 8 ++++++-- fs/nls/nls_cp932.c | 8 ++++++-- fs/nls/nls_cp936.c | 8 ++++++-- fs/nls/nls_cp949.c | 8 ++++++-- fs/nls/nls_cp950.c | 8 ++++++-- fs/nls/nls_euc-jp.c | 8 ++++++-- fs/nls/nls_iso8859-1.c | 8 ++++++-- fs/nls/nls_iso8859-13.c | 8 ++++++-- fs/nls/nls_iso8859-14.c | 8 ++++++-- fs/nls/nls_iso8859-15.c | 8 ++++++-- fs/nls/nls_iso8859-2.c | 8 ++++++-- fs/nls/nls_iso8859-3.c | 8 ++++++-- fs/nls/nls_iso8859-4.c | 8 ++++++-- fs/nls/nls_iso8859-5.c | 8 ++++++-- fs/nls/nls_iso8859-6.c | 8 ++++++-- fs/nls/nls_iso8859-7.c | 8 ++++++-- fs/nls/nls_iso8859-9.c | 8 ++++++-- fs/nls/nls_koi8-r.c | 8 ++++++-- fs/nls/nls_koi8-ru.c | 8 ++++++-- fs/nls/nls_koi8-u.c | 8 ++++++-- fs/nls/nls_utf8.c | 8 ++++++-- fs/udf/unicode.c | 4 ++-- include/linux/nls.h | 16 ++++++++++------ 54 files changed, 324 insertions(+), 112 deletions(-)
fs/nls/mac-celtic.c +6 -2
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macceltic" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-centeuro.c +6 -2
@@ -507,10 +507,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="maccenteuro" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-croatian.c +6 -2
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="maccroatian" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-cyrillic.c +6 -2
@@ -472,10 +472,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="maccyrillic" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-gaelic.c +6 -2
@@ -542,10 +542,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macgaelic" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-greek.c +6 -2
@@ -472,10 +472,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macgreek" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-iceland.c +6 -2
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="maciceland" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-inuit.c +6 -2
@@ -507,10 +507,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macinuit" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-roman.c +6 -2
@@ -612,10 +612,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macroman" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-romanian.c +6 -2
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macromanian" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/mac-turkish.c +6 -2
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="macturkish" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_ascii.c +6 -2
@@ -142,10 +142,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="ascii" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_base.c +6 -2
@@ -520,10 +520,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table default_table = { .charset ="default" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp1250.c +6 -2
@@ -323,10 +323,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp1250" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp1251.c +6 -2
@@ -277,10 +277,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp1251" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp1255.c +6 -2
@@ -358,11 +358,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp1255" , .alias ="iso8859-8" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp437.c +6 -2
@@ -363,10 +363,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp437" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp737.c +6 -2
@@ -326,10 +326,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp737" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp775.c +6 -2
@@ -295,10 +295,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp775" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp850.c +6 -2
@@ -291,10 +291,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp850" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp852.c +6 -2
@@ -313,10 +313,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp852" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp855.c +6 -2
@@ -275,10 +275,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp855" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp857.c +6 -2
@@ -277,10 +277,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp857" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp860.c +6 -2
@@ -340,10 +340,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp860" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp861.c +6 -2
@@ -363,10 +363,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp861" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp862.c +6 -2
@@ -397,10 +397,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp862" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp863.c +6 -2
@@ -357,10 +357,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp863" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp864.c +6 -2
@@ -383,10 +383,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp864" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp865.c +6 -2
@@ -363,10 +363,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp865" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp866.c +6 -2
@@ -281,10 +281,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp866" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp869.c +6 -2
@@ -291,10 +291,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp869" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp874.c +6 -2
@@ -249,11 +249,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp874" , .alias ="tis-620" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp932.c +6 -2
@@ -7907,11 +7907,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return -EINVAL ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp932" , .alias ="sjis" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp936.c +6 -2
@@ -11085,11 +11085,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp936" , .alias ="gb2312" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp949.c +6 -2
@@ -13920,11 +13920,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp949" , .alias ="euc-kr" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_cp950.c +6 -2
@@ -9456,11 +9456,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="cp950" , .alias ="big5" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_euc-jp.c +6 -2
@@ -549,10 +549,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return euc_offset; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="euc-jp" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, };static int __init init_nls_euc_jp (void )
fs/nls/nls_iso8859-1.c +6 -2
@@ -233,10 +233,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-1" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-13.c +6 -2
@@ -261,10 +261,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-13" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-14.c +6 -2
@@ -317,10 +317,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-14" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-15.c +6 -2
@@ -283,10 +283,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-15" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-2.c +6 -2
@@ -284,10 +284,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-2" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-3.c +6 -2
@@ -284,10 +284,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-3" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-4.c +6 -2
@@ -284,10 +284,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-4" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-5.c +6 -2
@@ -248,10 +248,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-5" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-6.c +6 -2
@@ -239,10 +239,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-6" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-7.c +6 -2
@@ -293,10 +293,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-7" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_iso8859-9.c +6 -2
@@ -248,10 +248,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="iso8859-9" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_koi8-r.c +6 -2
@@ -299,10 +299,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="koi8-r" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_koi8-ru.c +6 -2
@@ -51,10 +51,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="koi8-ru" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, };static int __init init_nls_koi8_ru (void )
fs/nls/nls_koi8-u.c +6 -2
@@ -306,10 +306,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="koi8-u" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, };
fs/nls/nls_utf8.c +6 -2
@@ -40,10 +40,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return n; } +static const struct nls_ops charset_ops = { + .uni2char = uni2char, + .char2uni = char2uni, +}; +static struct nls_table table = { .charset ="utf8" , - .uni2char = uni2char, - .char2uni = char2uni, + .ops = &charset_ops, .charset2lower = identity,/* no conversion */ .charset2upper = identity, };
fs/udf/unicode.c +2 -2
@@ -178,7 +178,7 @@ static int udf_name_from_CS0(struct super_block *sb, }if (UDF_QUERY_FLAG (sb,UDF_FLAG_NLS_MAP )) - conv_f =UDF_SB (sb)->s_nls_map->uni2char; + conv_f =UDF_SB (sb)->s_nls_map->ops->uni2char;else conv_f =NULL ;
@@ -286,7 +286,7 @@ static int udf_name_to_CS0(struct super_block *sb,return 0 ;if (UDF_QUERY_FLAG (sb,UDF_FLAG_NLS_MAP )) - conv_f =UDF_SB (sb)->s_nls_map->char2uni; + conv_f =UDF_SB (sb)->s_nls_map->ops->char2uni;else conv_f =NULL ;
include/linux/nls.h +10 -6
@@ -22,12 +22,16 @@ typedef u16 wchar_t;/* Arbitrary Unicode character */ typedef u32 unicode_t ; -struct nls_table { -const char *charset; -const char *alias; +struct nls_ops {int (*uni2char) (wchar_t uni,unsigned char *out,int boundlen);int (*char2uni) (const unsigned char *rawstring,int boundlen,wchar_t *uni); +}; + +struct nls_table { +const char *charset; +const char *alias; +const struct nls_ops *ops;const unsigned char *charset2lower;const unsigned char *charset2upper;struct module *owner;
@@ -62,14 +66,14 @@ extern int utf16s_to_utf8s(const wchar_t *pwcs, int len,static inline int nls_uni2char (const struct nls_table *table,wchar_t uni,unsigned char *out,int boundlen) { -return table->uni2char (uni, out, boundlen); +return table->ops->uni2char (uni, out, boundlen); }static inline int nls_char2uni (const struct nls_table *table,const unsigned char *rawstring,int boundlen,wchar_t *uni) { -return table->char2uni (rawstring, boundlen, uni); +return table->ops->char2uni (rawstring, boundlen, uni); }static inline const char *nls_charset_name (const struct nls_table *table)
@@ -116,7 +120,7 @@ nls_nullsize(const struct nls_table *codepage)int charlen;char tmp[NLS_MAX_CHARSET_SIZE ]; - charlen = codepage->uni2char (0 , tmp,NLS_MAX_CHARSET_SIZE ); + charlen = codepage->ops->uni2char (0 , tmp,NLS_MAX_CHARSET_SIZE );return charlen >0 ? charlen :1 ; }
Signature · 1 line
2.19.0
PATCH RESEND v2 4/25 nls: Split default charset from NLS core
In reply to Gabriel Krisman Bertazi---
fs/nls/Makefile | 1 +
fs/nls/nls_core.c | 94 ++++++++++++++++++++++++++++
fs/nls/{nls_base.c => nls_default.c} | 93 +++------------------------
3 files changed, 102 insertions(+), 86 deletions(-)
create mode 100644 fs/nls/nls_core.c
rename fs/nls/{nls_base.c => nls_default.c} (90%)fs/nls/Makefile +1 -0
@@ -3,6 +3,7 @@ # Makefile for native language support # +nls_base-y := nls_core.o nls_default.o obj-$(CONFIG_NLS) += nls_base.o obj-$(CONFIG_NLS_CODEPAGE_437) += nls_cp437.o
fs/nls/nls_core.c new file +94 -0
94 lines not on this page · raw message
fs/nls/nls_default.c renamed from fs/nls/nls_base.c +7 -86
118 lines not on this page · raw message
Signature · 1 line
2.19.0
PATCH RESEND v2 5/25 nls: Split struct nls_charset from struct nls_table
In reply to Gabriel Krisman BertaziIn order to support multiple versions of the same encoding, we want to split the charset registering data, which enrolls it with the NLS subsystem and is valid for all versions of the encoding, from the version specific data, that is actually going to be returned when the caller loads an NLS charset.
With the exception of the following files (files declaration and default charset), which were edited by hand, the other files were generated with the following Coccinelle patch:
Files edited by hand:
- fs/nls/nls_base.c
- include/linux/nls.h
- fs/nls/nls_default.c
<smpl>
@nlstable@
identifier p;
expression charset_str;
@@
static struct nls_table p = {
- .charset = charset_str,
};
@createops@
identifier nlstable.p;
expression nlstable.charset_str;
@@
+static struct nls_charset nls_charset;
static struct nls_table p = {
+ .charset = &nls_charset,
};
+
+static struct nls_charset nls_charset = {
+ .charset = charset_str,
+ .tables = &p,
+};
@@
expression A;
@@
? return
- register_nls(A);
+ register_nls(&nls_charset);
@@
expression A;
@@
- unregister_nls(A);
+ unregister_nls(&nls_charset);
@mvalias@
identifier p;
expression alias_str;
@@
static struct nls_table p = {
- .alias = alias_str,
};
@@
expression mvalias.alias_str;
@@
static struct nls_charset nls_charset = {
+ .alias = alias_str,
};
</smpl>
--- fs/nls/mac-celtic.c | 12 ++++++++--- fs/nls/mac-centeuro.c | 12 ++++++++--- fs/nls/mac-croatian.c | 12 ++++++++--- fs/nls/mac-cyrillic.c | 12 ++++++++--- fs/nls/mac-gaelic.c | 12 ++++++++--- fs/nls/mac-greek.c | 12 ++++++++--- fs/nls/mac-iceland.c | 12 ++++++++--- fs/nls/mac-inuit.c | 12 ++++++++--- fs/nls/mac-roman.c | 12 ++++++++--- fs/nls/mac-romanian.c | 12 ++++++++--- fs/nls/mac-turkish.c | 12 ++++++++--- fs/nls/nls_ascii.c | 12 ++++++++--- fs/nls/nls_core.c | 48 +++++++++++++++++++++++++++-------------- fs/nls/nls_cp1250.c | 12 ++++++++--- fs/nls/nls_cp1251.c | 12 ++++++++--- fs/nls/nls_cp1255.c | 14 ++++++++---- fs/nls/nls_cp437.c | 12 ++++++++--- fs/nls/nls_cp737.c | 12 ++++++++--- fs/nls/nls_cp775.c | 12 ++++++++--- fs/nls/nls_cp850.c | 12 ++++++++--- fs/nls/nls_cp852.c | 12 ++++++++--- fs/nls/nls_cp855.c | 12 ++++++++--- fs/nls/nls_cp857.c | 12 ++++++++--- fs/nls/nls_cp860.c | 12 ++++++++--- fs/nls/nls_cp861.c | 12 ++++++++--- fs/nls/nls_cp862.c | 12 ++++++++--- fs/nls/nls_cp863.c | 12 ++++++++--- fs/nls/nls_cp864.c | 12 ++++++++--- fs/nls/nls_cp865.c | 12 ++++++++--- fs/nls/nls_cp866.c | 12 ++++++++--- fs/nls/nls_cp869.c | 12 ++++++++--- fs/nls/nls_cp874.c | 14 ++++++++---- fs/nls/nls_cp932.c | 14 ++++++++---- fs/nls/nls_cp936.c | 14 ++++++++---- fs/nls/nls_cp949.c | 14 ++++++++---- fs/nls/nls_cp950.c | 14 ++++++++---- fs/nls/nls_default.c | 9 ++++++-- fs/nls/nls_euc-jp.c | 12 ++++++++--- fs/nls/nls_iso8859-1.c | 12 ++++++++--- fs/nls/nls_iso8859-13.c | 12 ++++++++--- fs/nls/nls_iso8859-14.c | 12 ++++++++--- fs/nls/nls_iso8859-15.c | 12 ++++++++--- fs/nls/nls_iso8859-2.c | 12 ++++++++--- fs/nls/nls_iso8859-3.c | 12 ++++++++--- fs/nls/nls_iso8859-4.c | 12 ++++++++--- fs/nls/nls_iso8859-5.c | 12 ++++++++--- fs/nls/nls_iso8859-6.c | 12 ++++++++--- fs/nls/nls_iso8859-7.c | 12 ++++++++--- fs/nls/nls_iso8859-9.c | 12 ++++++++--- fs/nls/nls_koi8-r.c | 12 ++++++++--- fs/nls/nls_koi8-ru.c | 12 ++++++++--- fs/nls/nls_koi8-u.c | 12 ++++++++--- fs/nls/nls_utf8.c | 12 ++++++++--- include/linux/nls.h | 18 ++++++++++------ 54 files changed, 516 insertions(+), 183 deletions(-)
fs/nls/mac-celtic.c +9 -3
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macceltic" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macceltic" , + .tables = &table, +}; +static int __init init_nls_macceltic (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macceltic(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macceltic)
fs/nls/mac-centeuro.c +9 -3
@@ -512,21 +512,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="maccenteuro" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="maccenteuro" , + .tables = &table, +}; +static int __init init_nls_maccenteuro (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_maccenteuro(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_maccenteuro)
fs/nls/mac-croatian.c +9 -3
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="maccroatian" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="maccroatian" , + .tables = &table, +}; +static int __init init_nls_maccroatian (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_maccroatian(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_maccroatian)
fs/nls/mac-cyrillic.c +9 -3
@@ -477,21 +477,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="maccyrillic" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="maccyrillic" , + .tables = &table, +}; +static int __init init_nls_maccyrillic (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_maccyrillic(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_maccyrillic)
fs/nls/mac-gaelic.c +9 -3
@@ -547,21 +547,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macgaelic" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macgaelic" , + .tables = &table, +}; +static int __init init_nls_macgaelic (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macgaelic(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macgaelic)
fs/nls/mac-greek.c +9 -3
@@ -477,21 +477,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macgreek" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macgreek" , + .tables = &table, +}; +static int __init init_nls_macgreek (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macgreek(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macgreek)
fs/nls/mac-iceland.c +9 -3
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="maciceland" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="maciceland" , + .tables = &table, +}; +static int __init init_nls_maciceland (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_maciceland(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_maciceland)
fs/nls/mac-inuit.c +9 -3
@@ -512,21 +512,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macinuit" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macinuit" , + .tables = &table, +}; +static int __init init_nls_macinuit (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macinuit(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macinuit)
fs/nls/mac-roman.c +9 -3
@@ -617,21 +617,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macroman" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macroman" , + .tables = &table, +}; +static int __init init_nls_macroman (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macroman(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macroman)
fs/nls/mac-romanian.c +9 -3
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macromanian" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macromanian" , + .tables = &table, +}; +static int __init init_nls_macromanian (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macromanian(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macromanian)
fs/nls/mac-turkish.c +9 -3
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="macturkish" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="macturkish" , + .tables = &table, +}; +static int __init init_nls_macturkish (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_macturkish(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_macturkish)
fs/nls/nls_ascii.c +9 -3
@@ -147,21 +147,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="ascii" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="ascii" , + .tables = &table, +}; +static int __init init_nls_ascii (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_ascii(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_ascii)
fs/nls/nls_core.c +32 -16
92 lines not on this page · raw message
fs/nls/nls_cp1250.c +9 -3
@@ -328,20 +328,26 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp1250" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp1250" , + .tables = &table, +}; +static int __init init_nls_cp1250 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp1250(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp1250)
fs/nls/nls_cp1251.c +9 -3
@@ -282,21 +282,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp1251" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp1251" , + .tables = &table, +}; +static int __init init_nls_cp1251 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp1251(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp1251)
fs/nls/nls_cp1255.c +10 -4
@@ -363,22 +363,28 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp1255" , - .alias ="iso8859-8" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .alias ="iso8859-8" , + .charset ="cp1255" , + .tables = &table, +}; +static int __init init_nls_cp1255 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp1255(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp1255)
fs/nls/nls_cp437.c +9 -3
@@ -368,21 +368,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp437" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp437" , + .tables = &table, +}; +static int __init init_nls_cp437 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp437(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp437)
fs/nls/nls_cp737.c +9 -3
@@ -331,21 +331,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp737" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp737" , + .tables = &table, +}; +static int __init init_nls_cp737 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp737(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp737)
fs/nls/nls_cp775.c +9 -3
@@ -300,21 +300,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp775" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp775" , + .tables = &table, +}; +static int __init init_nls_cp775 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp775(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp775)
fs/nls/nls_cp850.c +9 -3
@@ -296,21 +296,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp850" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp850" , + .tables = &table, +}; +static int __init init_nls_cp850 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp850(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp850)
fs/nls/nls_cp852.c +9 -3
@@ -318,21 +318,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp852" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp852" , + .tables = &table, +}; +static int __init init_nls_cp852 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp852(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp852)
fs/nls/nls_cp855.c +9 -3
@@ -280,21 +280,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp855" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp855" , + .tables = &table, +}; +static int __init init_nls_cp855 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp855(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp855)
fs/nls/nls_cp857.c +9 -3
@@ -282,21 +282,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp857" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp857" , + .tables = &table, +}; +static int __init init_nls_cp857 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp857(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp857)
fs/nls/nls_cp860.c +9 -3
@@ -345,21 +345,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp860" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp860" , + .tables = &table, +}; +static int __init init_nls_cp860 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp860(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp860)
fs/nls/nls_cp861.c +9 -3
@@ -368,21 +368,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp861" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp861" , + .tables = &table, +}; +static int __init init_nls_cp861 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp861(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp861)
fs/nls/nls_cp862.c +9 -3
@@ -402,21 +402,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp862" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp862" , + .tables = &table, +}; +static int __init init_nls_cp862 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp862(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp862)
fs/nls/nls_cp863.c +9 -3
@@ -362,21 +362,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp863" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp863" , + .tables = &table, +}; +static int __init init_nls_cp863 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp863(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp863)
fs/nls/nls_cp864.c +9 -3
@@ -388,21 +388,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp864" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp864" , + .tables = &table, +}; +static int __init init_nls_cp864 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp864(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp864)
fs/nls/nls_cp865.c +9 -3
@@ -368,21 +368,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp865" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp865" , + .tables = &table, +}; +static int __init init_nls_cp865 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp865(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp865)
fs/nls/nls_cp866.c +9 -3
@@ -286,21 +286,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp866" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp866" , + .tables = &table, +}; +static int __init init_nls_cp866 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp866(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp866)
fs/nls/nls_cp869.c +9 -3
@@ -296,21 +296,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp869" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="cp869" , + .tables = &table, +}; +static int __init init_nls_cp869 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp869(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp869)
fs/nls/nls_cp874.c +10 -4
@@ -254,22 +254,28 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp874" , - .alias ="tis-620" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .alias ="tis-620" , + .charset ="cp874" , + .tables = &table, +}; +static int __init init_nls_cp874 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp874(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp874)
fs/nls/nls_cp932.c +10 -4
@@ -7912,22 +7912,28 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp932" , - .alias ="sjis" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .alias ="sjis" , + .charset ="cp932" , + .tables = &table, +}; +static int __init init_nls_cp932 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp932(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp932)
fs/nls/nls_cp936.c +10 -4
@@ -11090,22 +11090,28 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp936" , - .alias ="gb2312" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .alias ="gb2312" , + .charset ="cp936" , + .tables = &table, +}; +static int __init init_nls_cp936 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp936(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp936)
fs/nls/nls_cp949.c +10 -4
@@ -13925,22 +13925,28 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp949" , - .alias ="euc-kr" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .alias ="euc-kr" , + .charset ="cp949" , + .tables = &table, +}; +static int __init init_nls_cp949 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp949(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp949)
fs/nls/nls_cp950.c +10 -4
@@ -9461,22 +9461,28 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="cp950" , - .alias ="big5" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .alias ="big5" , + .charset ="cp950" , + .tables = &table, +}; +static int __init init_nls_cp950 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_cp950(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_cp950)
fs/nls/nls_default.c +7 -2
@@ -17,7 +17,7 @@#include <asm/byteorder.h> #include <linux/nls.h> -static struct nls_table default_table; +struct nls_charset default_charset;struct utf8_table {int cmask;
@@ -453,12 +453,17 @@ static const struct nls_ops charset_ops = { };static struct nls_table default_table = { - .charset ="default" , + .charset = &default_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +struct nls_charset default_charset = { + .charset ="default" , + .tables = &default_table, +}; +/* Returns a simple default translation table */ struct nls_table *load_nls_default (void ) {
fs/nls/nls_euc-jp.c +9 -3
@@ -554,11 +554,17 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="euc-jp" , + .charset = &nls_charset, .ops = &charset_ops, }; +static struct nls_charset nls_charset = { + .charset ="euc-jp" , + .tables = &table, +}; +static int __init init_nls_euc_jp (void ) { p_nls =load_nls ("cp932" );
@@ -566,7 +572,7 @@ static int __init init_nls_euc_jp(void)if (p_nls) { table.charset2upper = p_nls->charset2upper; table.charset2lower = p_nls->charset2lower; -return register_nls (&table); +return register_nls (&nls_charset); }return -EINVAL ;
@@ -574,7 +580,7 @@ static int __init init_nls_euc_jp(void) static void __exit exit_nls_euc_jp(void) { -unregister_nls (&table); +unregister_nls (&nls_charset);unload_nls (p_nls); }
fs/nls/nls_iso8859-1.c +9 -3
@@ -238,21 +238,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-1" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-1" , + .tables = &table, +}; +static int __init init_nls_iso8859_1 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_1(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_1)
fs/nls/nls_iso8859-13.c +9 -3
@@ -266,21 +266,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-13" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-13" , + .tables = &table, +}; +static int __init init_nls_iso8859_13 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_13(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_13)
fs/nls/nls_iso8859-14.c +9 -3
@@ -322,21 +322,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-14" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-14" , + .tables = &table, +}; +static int __init init_nls_iso8859_14 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_14(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_14)
fs/nls/nls_iso8859-15.c +9 -3
@@ -288,21 +288,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-15" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-15" , + .tables = &table, +}; +static int __init init_nls_iso8859_15 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_15(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_15)
fs/nls/nls_iso8859-2.c +9 -3
@@ -289,21 +289,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-2" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-2" , + .tables = &table, +}; +static int __init init_nls_iso8859_2 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_2(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_2)
fs/nls/nls_iso8859-3.c +9 -3
@@ -289,21 +289,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-3" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-3" , + .tables = &table, +}; +static int __init init_nls_iso8859_3 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_3(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_3)
fs/nls/nls_iso8859-4.c +9 -3
@@ -289,21 +289,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-4" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-4" , + .tables = &table, +}; +static int __init init_nls_iso8859_4 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_4(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_4)
fs/nls/nls_iso8859-5.c +9 -3
@@ -253,21 +253,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-5" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-5" , + .tables = &table, +}; +static int __init init_nls_iso8859_5 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_5(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_5)
fs/nls/nls_iso8859-6.c +9 -3
@@ -244,21 +244,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-6" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-6" , + .tables = &table, +}; +static int __init init_nls_iso8859_6 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_6(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_6)
fs/nls/nls_iso8859-7.c +9 -3
@@ -298,21 +298,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-7" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-7" , + .tables = &table, +}; +static int __init init_nls_iso8859_7 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_7(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_7)
fs/nls/nls_iso8859-9.c +9 -3
@@ -253,21 +253,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="iso8859-9" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="iso8859-9" , + .tables = &table, +}; +static int __init init_nls_iso8859_9 (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_iso8859_9(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_iso8859_9)
fs/nls/nls_koi8-r.c +9 -3
@@ -304,21 +304,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="koi8-r" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="koi8-r" , + .tables = &table, +}; +static int __init init_nls_koi8_r (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_koi8_r(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_koi8_r)
fs/nls/nls_koi8-ru.c +9 -3
@@ -56,11 +56,17 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="koi8-ru" , + .charset = &nls_charset, .ops = &charset_ops, }; +static struct nls_charset nls_charset = { + .charset ="koi8-ru" , + .tables = &table, +}; +static int __init init_nls_koi8_ru (void ) { p_nls =load_nls ("koi8-u" );
@@ -68,7 +74,7 @@ static int __init init_nls_koi8_ru(void)if (p_nls) { table.charset2upper = p_nls->charset2upper; table.charset2lower = p_nls->charset2lower; -return register_nls (&table); +return register_nls (&nls_charset); }return -EINVAL ;
@@ -76,7 +82,7 @@ static int __init init_nls_koi8_ru(void) static void __exit exit_nls_koi8_ru(void) { -unregister_nls (&table); +unregister_nls (&nls_charset);unload_nls (p_nls); }
fs/nls/nls_koi8-u.c +9 -3
@@ -311,21 +311,27 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="koi8-u" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = charset2lower, .charset2upper = charset2upper, }; +static struct nls_charset nls_charset = { + .charset ="koi8-u" , + .tables = &table, +}; +static int __init init_nls_koi8_u (void ) { -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_koi8_u(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_koi8_u)
fs/nls/nls_utf8.c +9 -3
@@ -45,25 +45,31 @@ static const struct nls_ops charset_ops = { .char2uni = char2uni, }; +static struct nls_charset nls_charset;static struct nls_table table = { - .charset ="utf8" , + .charset = &nls_charset, .ops = &charset_ops, .charset2lower = identity,/* no conversion */ .charset2upper = identity, }; +static struct nls_charset nls_charset = { + .charset ="utf8" , + .tables = &table, +}; +static int __init init_nls_utf8 (void ) {int i;for (i=0 ; i<256 ; i++) identity[i] = i; -return register_nls (&table); +return register_nls (&nls_charset); } static void __exit exit_nls_utf8(void) { -unregister_nls (&table); +unregister_nls (&nls_charset); }module_init (init_nls_utf8)
include/linux/nls.h +12 -6
@@ -29,15 +29,21 @@ struct nls_ops { };struct nls_table { -const char *charset; -const char *alias; +const struct nls_charset *charset;const struct nls_ops *ops;const unsigned char *charset2lower;const unsigned char *charset2upper; -struct module *owner;struct nls_table *next; }; +struct nls_charset { +const char *charset; +const char *alias; +struct module *owner; +struct nls_table *tables; +struct nls_charset *next; +}; +/* this value hold the maximum octet of charset */ #define NLS_MAX_CHARSET_SIZE 6/* for UTF-8 */
@@ -49,8 +55,8 @@ enum utf16_endian { };/* nls_base.c */ -extern int __register_nls (struct nls_table *,struct module *); -extern int unregister_nls (struct nls_table *); +extern int __register_nls (struct nls_charset *,struct module *); +extern int unregister_nls (struct nls_charset *);extern struct nls_table *load_nls (char *);extern void unload_nls (struct nls_table *);extern struct nls_table *load_nls_default (void );
@@ -78,7 +84,7 @@ static inline int nls_char2uni(const struct nls_table *table,static inline const char *nls_charset_name (const struct nls_table *table) { -return table->charset; +return table->charset->charset; }static inline unsigned char nls_tolower (struct nls_table *t,unsigned char c)
Signature · 1 line
2.19.0
PATCH RESEND v2 6/25 nls: Add support for multiple versions of an encoding
In reply to Gabriel Krisman BertaziThis allows a user to request a specific version of an encoding, like the version 10.0.0 of Unicode encoded in utf8.
Supporting specific versions of encodings is important to ensure stability of names in filesystems, specially when doing transformations like casefold and normalization. Even for unicode, where defined code-points are stable, there is instability for code points that weren't defined on a previous version, so the user might want to use an older version of the encoding to ensure the encoding is exact.
Not every NLS charset supports this feature. It doesn't make sense for many of them, like ASCII. Others just don't implement it yet, and never will. In those cases, the interface allows the caller to get the un-versioned charset, which is the same original behavior as if this patch weren't applied. A user that is not interested in a specific version can also ask for a versioned charset without specifying the version, and in this case, NLS will return the latest version available of that charset.
--- fs/nls/nls_core.c | 45 ++++++++++++++++++++++++++++++++++++++------- include/linux/nls.h | 8 ++++++++ 2 files changed, 46 insertions(+), 7 deletions(-)
fs/nls/nls_core.c +38 -7
@@ -19,10 +19,26 @@extern struct nls_charset default_charset;static struct nls_charset *charsets = &default_charset;static DEFINE_SPINLOCK (nls_lock); -static struct nls_table *nls_load_table (struct nls_charset *charset) + +static struct nls_table *nls_load_table (struct nls_charset *charset, +const char *version, +unsigned int flags) { -/* For now, return the default table, which is the first one found. */ -return charset->tables; +struct nls_table *tbl; + +/* If there is no table_create, only 1 table is supported and it +* must have been loaded statically. +*/ +if (!charset->load_table) + tbl = charset->tables; +else + tbl = charset->load_table (version, flags); + +if (IS_ERR (tbl)) +return tbl; + + tbl->flags = flags; +return tbl; }int __register_nls (struct nls_charset *nls,struct module *owner)
@@ -85,21 +101,36 @@ static struct nls_charset *find_nls(const char *charset)return nls; } -struct nls_table *load_nls (char *charset) +struct nls_table *load_nls_version (const char *charset,const char *version, +unsigned int flags) {struct nls_charset *nls_charset; nls_charset =try_then_request_module (find_nls (charset),"nls_%s" , charset); -if (!IS_ERR (nls_charset)) +if (IS_ERR (nls_charset)) +return ERR_PTR (-EINVAL ); + +return nls_load_table (nls_charset, version, flags); +} +EXPORT_SYMBOL (load_nls_version); + +struct nls_table *load_nls (char *charset) +{ +struct nls_table *table =load_nls_version (charset,NULL ,0 ); + +/* Pre-versioned load_nls() didn't return error pointers. Let's +* keep the abi for now to prevent breakage. +*/ +if (IS_ERR (table))return NULL ; -return nls_load_table (nls_charset); +return table; }void unload_nls (struct nls_table *nls) { -if (nls) +if (!IS_ERR_OR_NULL (nls))module_put (nls->charset->owner); }
include/linux/nls.h +8 -0
@@ -30,6 +30,9 @@ struct nls_ops {struct nls_table {const struct nls_charset *charset; +unsigned int version; +unsigned int flags; +const struct nls_ops *ops;const unsigned char *charset2lower;const unsigned char *charset2upper;
@@ -42,6 +45,8 @@ struct nls_charset {struct module *owner;struct nls_table *tables;struct nls_charset *next; +struct nls_table *(*load_table)(const char *version, +unsigned int flags); };/* this value hold the maximum octet of charset */
@@ -58,6 +63,9 @@ enum utf16_endian {extern int __register_nls (struct nls_charset *,struct module *);extern int unregister_nls (struct nls_charset *);extern struct nls_table *load_nls (char *); +extern struct nls_table *load_nls_version (const char *charset, +const char *version, +unsigned int flags);extern void unload_nls (struct nls_table *);extern struct nls_table *load_nls_default (void );#define register_nls (nls) __register_nls((nls), THIS_MODULE)
Signature · 1 line
2.19.0
PATCH RESEND v2 7/25 nls: Implement NLS_STRICT_MODE flag
In reply to Gabriel Krisman BertaziThe flag NLS_STRICT_MODE indicates whether NLS should reject invalid characters or ignore them. Support for this relies on the .validate() hook, which is implemented by each charset and states whether a given string is valid within that charset.
--- fs/nls/nls_core.c | 11 +++++++++++ include/linux/nls.h | 25 +++++++++++++++++++++++++ 2 files changed, 36 insertions(+)
fs/nls/nls_core.c +11 -0
@@ -20,6 +20,14 @@ extern struct nls_charset default_charset;static struct nls_charset *charsets = &default_charset;static DEFINE_SPINLOCK (nls_lock); +static int nls_validate_flags (struct nls_table *table,unsigned int flags) +{ +if (flags &NLS_STRICT_MODE && !table->ops->validate) +return -1 ; + +return 0 ; +} +static struct nls_table *nls_load_table (struct nls_charset *charset,const char *version,unsigned int flags)
@@ -37,6 +45,9 @@ static struct nls_table *nls_load_table(struct nls_charset *charset,if (IS_ERR (tbl))return tbl; +if (nls_validate_flags (tbl, flags) <0 ) +return ERR_PTR (-EINVAL ); + tbl->flags = flags;return tbl; }
include/linux/nls.h +25 -0
@@ -22,10 +22,22 @@ typedef u16 wchar_t;/* Arbitrary Unicode character */ typedef u32 unicode_t ; +struct nls_table ; +struct nls_ops {int (*uni2char) (wchar_t uni,unsigned char *out,int boundlen);int (*char2uni) (const unsigned char *rawstring,int boundlen,wchar_t *uni); +/** +* @validate: +* +* Returns 0 if the argument is a valid string in this charset. +* Otherwise, return non-zero. +* +* This is required iff the charset supports strict mode. +**/ +int (*validate)(const struct nls_table *charset, +const unsigned char *str,size_t len); };struct nls_table {
@@ -59,6 +71,13 @@ enum utf16_endian { UTF16_BIG_ENDIAN }; +#define NLS_STRICT_MODE 0x00000001 + +static inline int IS_STRICT_MODE (const struct nls_table *charset) +{ +return (charset->flags &NLS_STRICT_MODE ); +} +/* nls_base.c */ extern int __register_nls (struct nls_charset *,struct module *);extern int unregister_nls (struct nls_charset *);
@@ -90,6 +109,12 @@ static inline int nls_char2uni(const struct nls_table *table,return table->ops->char2uni (rawstring, boundlen, uni); } +static inline int nls_validate (const struct nls_table *t,const unsigned char *str, +const size_t len) +{ +return t->ops->validate (t, str, len); +} +static inline const char *nls_charset_name (const struct nls_table *table) {return table->charset->charset;
Signature · 1 line
2.19.0
PATCH RESEND v2 8/25 nls: Let charsets define the behavior of tolower/toupper
In reply to Gabriel Krisman BertaziInstead of always reading from a table, give the charset a chance to implement tolower() and toupper() algorithmically.
This allow us to drop a lot of tables which hardcode the identity functions (like ASCII), and replace them with a few lines of code in the hooks.
This patch was created using the semantic patch below, with the exception of the header files (hook definitions) and a fix to files that didn't have the tables statically allocated (koi8-u and cp932).
<smpl>
@tbl@
identifier p;
expression lower_tbl;
expression upper_tbl;
@@
static struct nls_table p = {
- .charset2lower = lower_tbl,
- .charset2upper = upper_tbl,
};
@@
identifier charset_ops;
expression tbl.lower_tbl;
expression tbl.upper_tbl;
@@
+ static unsigned char charset_tolower(const struct nls_table *table, unsigned int c)
+ {
+ return lower_tbl[c];
+ }
+
+ static unsigned char charset_toupper(const struct nls_table *table, unsigned int c)
+ {
+ return upper_tbl[c];
+ }
static const struct nls_ops charset_ops = {
+ .lowercase = charset_toupper,
+ .uppercase = charset_tolower,
};
@@
struct nls_table *t;
expression A;
expression nc;
@@
(
- nc = t->charset2lower[A]
+ nc = nls_tolower(t, A)
|
- nc = t->charset2upper[A]
+ nc = nls_toupper(t, A)
)
<...
- if(!nc)
- nc = A;
...>
</smpl>
--- fs/fat/dir.c | 5 +---- fs/nls/mac-celtic.c | 14 ++++++++++++-- fs/nls/mac-centeuro.c | 14 ++++++++++++-- fs/nls/mac-croatian.c | 14 ++++++++++++-- fs/nls/mac-cyrillic.c | 14 ++++++++++++-- fs/nls/mac-gaelic.c | 14 ++++++++++++-- fs/nls/mac-greek.c | 14 ++++++++++++-- fs/nls/mac-iceland.c | 14 ++++++++++++-- fs/nls/mac-inuit.c | 14 ++++++++++++-- fs/nls/mac-roman.c | 14 ++++++++++++-- fs/nls/mac-romanian.c | 14 ++++++++++++-- fs/nls/mac-turkish.c | 14 ++++++++++++-- fs/nls/nls_ascii.c | 14 ++++++++++++-- fs/nls/nls_cp1250.c | 14 ++++++++++++-- fs/nls/nls_cp1251.c | 14 ++++++++++++-- fs/nls/nls_cp1255.c | 14 ++++++++++++-- fs/nls/nls_cp437.c | 14 ++++++++++++-- fs/nls/nls_cp737.c | 14 ++++++++++++-- fs/nls/nls_cp775.c | 14 ++++++++++++-- fs/nls/nls_cp850.c | 14 ++++++++++++-- fs/nls/nls_cp852.c | 14 ++++++++++++-- fs/nls/nls_cp855.c | 14 ++++++++++++-- fs/nls/nls_cp857.c | 14 ++++++++++++-- fs/nls/nls_cp860.c | 14 ++++++++++++-- fs/nls/nls_cp861.c | 14 ++++++++++++-- fs/nls/nls_cp862.c | 14 ++++++++++++-- fs/nls/nls_cp863.c | 14 ++++++++++++-- fs/nls/nls_cp864.c | 14 ++++++++++++-- fs/nls/nls_cp865.c | 14 ++++++++++++-- fs/nls/nls_cp866.c | 14 ++++++++++++-- fs/nls/nls_cp869.c | 14 ++++++++++++-- fs/nls/nls_cp874.c | 14 ++++++++++++-- fs/nls/nls_cp932.c | 14 ++++++++++++-- fs/nls/nls_cp936.c | 14 ++++++++++++-- fs/nls/nls_cp949.c | 14 ++++++++++++-- fs/nls/nls_cp950.c | 14 ++++++++++++-- fs/nls/nls_default.c | 14 ++++++++++++-- fs/nls/nls_euc-jp.c | 7 ++++--- fs/nls/nls_iso8859-1.c | 14 ++++++++++++-- fs/nls/nls_iso8859-13.c | 14 ++++++++++++-- fs/nls/nls_iso8859-14.c | 14 ++++++++++++-- fs/nls/nls_iso8859-15.c | 14 ++++++++++++-- fs/nls/nls_iso8859-2.c | 14 ++++++++++++-- fs/nls/nls_iso8859-3.c | 14 ++++++++++++-- fs/nls/nls_iso8859-4.c | 14 ++++++++++++-- fs/nls/nls_iso8859-5.c | 14 ++++++++++++-- fs/nls/nls_iso8859-6.c | 14 ++++++++++++-- fs/nls/nls_iso8859-7.c | 14 ++++++++++++-- fs/nls/nls_iso8859-9.c | 14 ++++++++++++-- fs/nls/nls_koi8-r.c | 14 ++++++++++++-- fs/nls/nls_koi8-ru.c | 6 +++--- fs/nls/nls_koi8-u.c | 14 ++++++++++++-- fs/nls/nls_utf8.c | 14 ++++++++++++-- include/linux/nls.h | 17 +++++++++++------ 54 files changed, 619 insertions(+), 116 deletions(-)
fs/fat/dir.c +1 -4
@@ -215,10 +215,7 @@ fat_short2lower_uni(struct nls_table *t, unsigned char *c, *uni =0x003f ;/* a question mark */ charlen =1 ; }else if (charlen <=1 ) { -unsigned char nc = t->charset2lower[*c]; - -if (!nc) - nc = *c; +unsigned char nc =nls_tolower (t, *c); charlen =nls_char2uni (t, &nc,1 , uni);if (charlen <0 ) {
fs/nls/mac-celtic.c +12 -2
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-centeuro.c +12 -2
@@ -507,7 +507,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -516,8 +528,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-croatian.c +12 -2
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-cyrillic.c +12 -2
@@ -472,7 +472,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -481,8 +493,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-gaelic.c +12 -2
@@ -542,7 +542,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -551,8 +563,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-greek.c +12 -2
@@ -472,7 +472,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -481,8 +493,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-iceland.c +12 -2
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-inuit.c +12 -2
@@ -507,7 +507,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -516,8 +528,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-roman.c +12 -2
@@ -612,7 +612,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -621,8 +633,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-romanian.c +12 -2
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/mac-turkish.c +12 -2
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_ascii.c +12 -2
@@ -142,7 +142,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -151,8 +163,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp1250.c +12 -2
@@ -323,7 +323,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -332,8 +344,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp1251.c +12 -2
@@ -277,7 +277,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -286,8 +298,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp1255.c +12 -2
@@ -358,7 +358,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -367,8 +379,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp437.c +12 -2
@@ -363,7 +363,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -372,8 +384,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp737.c +12 -2
@@ -326,7 +326,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -335,8 +347,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp775.c +12 -2
@@ -295,7 +295,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -304,8 +316,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp850.c +12 -2
@@ -291,7 +291,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -300,8 +312,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp852.c +12 -2
@@ -313,7 +313,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -322,8 +334,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp855.c +12 -2
@@ -275,7 +275,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -284,8 +296,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp857.c +12 -2
@@ -277,7 +277,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -286,8 +298,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp860.c +12 -2
@@ -340,7 +340,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -349,8 +361,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp861.c +12 -2
@@ -363,7 +363,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -372,8 +384,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp862.c +12 -2
@@ -397,7 +397,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -406,8 +418,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp863.c +12 -2
@@ -357,7 +357,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -366,8 +378,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp864.c +12 -2
@@ -383,7 +383,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -392,8 +404,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp865.c +12 -2
@@ -363,7 +363,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -372,8 +384,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp866.c +12 -2
@@ -281,7 +281,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -290,8 +302,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp869.c +12 -2
@@ -291,7 +291,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -300,8 +312,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp874.c +12 -2
@@ -249,7 +249,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -258,8 +270,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp932.c +12 -2
@@ -7907,7 +7907,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return -EINVAL ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -7916,8 +7928,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp936.c +12 -2
@@ -11085,7 +11085,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -11094,8 +11106,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp949.c +12 -2
@@ -13920,7 +13920,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -13929,8 +13941,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_cp950.c +12 -2
@@ -9456,7 +9456,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -9465,8 +9477,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_default.c +12 -2
@@ -447,7 +447,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -455,8 +467,6 @@ static const struct nls_ops charset_ops = {static struct nls_table default_table = { .charset = &default_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };struct nls_charset default_charset = {
fs/nls/nls_euc-jp.c +4 -3
@@ -549,7 +549,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return euc_offset; } -static const struct nls_ops charset_ops = { +static struct nls_ops charset_ops = { .uni2char = uni2char, .char2uni = char2uni, };
@@ -570,8 +570,9 @@ static int __init init_nls_euc_jp(void) p_nls =load_nls ("cp932" );if (p_nls) { - table.charset2upper = p_nls->charset2upper; - table.charset2lower = p_nls->charset2lower; + + charset_ops.uppercase = p_nls->ops->uppercase; + charset_ops.lowercase = p_nls->ops->lowercase;return register_nls (&nls_charset); }
fs/nls/nls_iso8859-1.c +12 -2
@@ -233,7 +233,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -242,8 +254,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-13.c +12 -2
@@ -261,7 +261,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -270,8 +282,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-14.c +12 -2
@@ -317,7 +317,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -326,8 +338,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-15.c +12 -2
@@ -283,7 +283,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -292,8 +304,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-2.c +12 -2
@@ -284,7 +284,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -293,8 +305,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-3.c +12 -2
@@ -284,7 +284,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -293,8 +305,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-4.c +12 -2
@@ -284,7 +284,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -293,8 +305,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-5.c +12 -2
@@ -248,7 +248,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -257,8 +269,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-6.c +12 -2
@@ -239,7 +239,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -248,8 +260,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-7.c +12 -2
@@ -293,7 +293,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -302,8 +314,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-9.c +12 -2
@@ -248,7 +248,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -257,8 +269,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_koi8-r.c +12 -2
@@ -299,7 +299,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -308,8 +320,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_koi8-ru.c +3 -3
@@ -51,7 +51,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,return n; } -static const struct nls_ops charset_ops = { +static struct nls_ops charset_ops = { .uni2char = uni2char, .char2uni = char2uni, };
@@ -72,8 +72,8 @@ static int __init init_nls_koi8_ru(void) p_nls =load_nls ("koi8-u" );if (p_nls) { - table.charset2upper = p_nls->charset2upper; - table.charset2lower = p_nls->charset2lower; + charset_ops.uppercase = p_nls->ops->uppercase; + charset_ops.lowercase = p_nls->ops->lowercase;return register_nls (&nls_charset); }
fs/nls/nls_koi8-u.c +12 -2
@@ -306,7 +306,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return charset2lower[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return charset2upper[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -315,8 +327,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = charset2lower, - .charset2upper = charset2upper, };static struct nls_charset nls_charset = {
fs/nls/nls_utf8.c +12 -2
@@ -40,7 +40,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return n; } +static unsigned char charset_tolower (const struct nls_table *table, +unsigned int c){ +return identity[c]; +} + +static unsigned char charset_toupper (const struct nls_table *table, +unsigned int c) { +return identity[c]; +} +static const struct nls_ops charset_ops = { + .lowercase = charset_toupper, + .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, };
@@ -49,8 +61,6 @@ static struct nls_charset nls_charset;static struct nls_table table = { .charset = &nls_charset, .ops = &charset_ops, - .charset2lower = identity,/* no conversion */ - .charset2upper = identity, };static struct nls_charset nls_charset = {
include/linux/nls.h +11 -6
@@ -38,6 +38,10 @@ struct nls_ops {**/ int (*validate)(const struct nls_table *charset,const unsigned char *str,size_t len); +unsigned char (*lowercase)(const struct nls_table *charset, +unsigned int c); +unsigned char (*uppercase)(const struct nls_table *charset, +unsigned int c); };struct nls_table {
@@ -46,9 +50,8 @@ struct nls_table {unsigned int flags;const struct nls_ops *ops; -const unsigned char *charset2lower; -const unsigned char *charset2upper;struct nls_table *next; + };struct nls_charset {
@@ -120,16 +123,18 @@ static inline const char *nls_charset_name(const struct nls_table *table)return table->charset->charset; } -static inline unsigned char nls_tolower (struct nls_table *t,unsigned char c) +static inline unsigned char nls_tolower (const struct nls_table *t, +unsigned char c) { -unsigned char nc = t->charset2lower[c]; +unsigned char nc = t->ops->lowercase (t, c);return nc ? nc : c; } -static inline unsigned char nls_toupper (struct nls_table *t,unsigned char c) +static inline unsigned char nls_toupper (const struct nls_table *t, +unsigned char c) { -unsigned char nc = t->charset2upper[c]; +unsigned char nc = t->ops->uppercase (t, c);return nc ? nc : c; }
Signature · 1 line
2.19.0
PATCH RESEND v2 9/25 nls: Add new interface for string comparisons
In reply to Gabriel Krisman BertaziThe existing stricmp() interface is limited by not accepting separated length parameters for each string being compared. This is a problem for charsets doing normalization or full casefold comparison, since different sized strings can still be matched. To resolve this problem, this patch implements a new interface, allowing charsets to do the comparison, if needed.
The original stricmp is left in the code, until we convert all caller to the new interface. Nevertheless, it was reimplemented using the new interface.
--- include/linux/nls.h | 69 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 3 deletions(-)
include/linux/nls.h +66 -3
@@ -3,6 +3,7 @@#define _LINUX_NLS_H#include <linux/init.h> +#include <linux/string.h> /* Unicode has changed over the years. Unicode code points no longer * fit into 16 bits; as of Unicode 5 valid code points range from 0
@@ -38,6 +39,32 @@ struct nls_ops {**/ int (*validate)(const struct nls_table *charset,const unsigned char *str,size_t len); +/** +* @strncmp: +* +* strncmp is the function for case-sensitive string comparison. +* It only needs to be implemented by charsets that want to do +* some fancy comparisons, like normalization-insensitive. +* +* Returns 0 if str1 and str2 are equal, otherwise return +* non-zero. +**/ +int (*strncmp)(const struct nls_table *charset, +const unsigned char *str1,size_t len1, +const unsigned char *str2,size_t len2); + +/** +* @strncasecmp: +* +* strncasecmp is the function for case-insensitive string +* comparison. +* +* Returns 0 if str1 and str2 are equal, otherwise return +* non-zero. +**/ +int (*strncasecmp)(const struct nls_table *charset, +const unsigned char *str1,size_t len1, +const unsigned char *str2,size_t len2);unsigned char (*lowercase)(const struct nls_table *charset,unsigned int c); unsigned char (*uppercase)(const struct nls_table *charset,
@@ -139,10 +166,21 @@ static inline unsigned char nls_toupper(const struct nls_table *t,return nc ? nc : c; } -static inline int nls_strnicmp (struct nls_table *t,const unsigned char *s1, -const unsigned char *s2,int len) +static inline int nls_strncasecmp (struct nls_table *t, +const unsigned char *s1,size_t len1, +const unsigned char *s2,size_t len2) { -while (len--) { +if (t->ops->strncasecmp) +return t->ops->strncasecmp (t, s1, len1, s2, len2); + +if (IS_STRICT_MODE (t) && + (nls_validate (t, s1, len1) ||nls_validate (t, s1, len1))) +return -EINVAL ; + +if (len1 != len2) +return 1 ; + +while (len1--) {if (nls_tolower (t, *s1++) !=nls_tolower (t, *s2++))return 1 ; }
@@ -150,6 +188,31 @@ static inline int nls_strnicmp(struct nls_table *t, const unsigned char *s1,return 0 ; } +static inline int nls_strncmp (struct nls_table *t, +const unsigned char *s1,size_t len1, +const unsigned char *s2,size_t len2) +{ +if (t->ops->strncmp) +return t->ops->strncmp (t, s1, len1, s2, len2); + +if (IS_STRICT_MODE (t) && + (nls_validate (t, s1, len1) ||nls_validate (t, s1, len1))) +return -EINVAL ; + +if (len1 != len2) +return 1 ; + +/* strnicmp did not return negative values. So let's keep the +* abi for now */ +return !!memcmp (s1, s2, len1); +} + +static inline int nls_strnicmp (struct nls_table *t,const unsigned char *s1, +const unsigned char *s2,int len) +{ +return nls_strncasecmp (t, s1, len, s2, len); +} +/* * nls_nullsize - return length of null character for codepage * @codepage - codepage for which to return length of NULL terminator
Signature · 1 line
2.19.0
PATCH RESEND v2 10/25 nls: Add optional normalization and casefold hooks
In reply to Gabriel Krisman BertaziThe Normalization operation applies a transformation to strings to obtain the normalization form, which allow the user to determine whether any two strings are equivalent to each other. The NLS subsystem doesn't impose any constraint on what means to be equivalent, for any charsets. Unicode-based charsets, for instance, are free to support one, a few or all kinds of Unicode equivalences.
The Casefold operation is similar to Normalization, in a sense that it also allows the caller to identify equivalent strings, but it disregards case, making it ideal for case insensitive comparisons.
Default implementation are provided by the nls core, such that existing charsets can operate on the new interface. The Normalization default operation is the format NLS_NORMALIZATION_TYPE_PLAIN, which returns the identity of the string, which means no normalization. The casefold default is NLS_CASEFOLD_TYPE_TOUPPER, which returns the string with all characters converted to uppercase.
Changes since V1:
- Add default operations for casefold and normalization
--- fs/nls/nls_core.c | 11 +++++ include/linux/nls.h | 116 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+)
fs/nls/nls_core.c +11 -0
@@ -25,6 +25,17 @@ static int nls_validate_flags(struct nls_table *table, unsigned int flags)if (flags &NLS_STRICT_MODE && !table->ops->validate)return -1 ; +if ((flags &NLS_NORMALIZATION_TYPE_MASK ) && !table->ops->normalize) +return -1 ; + +if ((flags &NLS_CASEFOLD_TYPE_MASK ) && !table->ops->casefold) +return -1 ; + +/* Reject unused flags */ +if (flags & ~(NLS_CASEFOLD_TYPE_MASK |NLS_NORMALIZATION_TYPE_MASK | +NLS_STRICT_MODE )) +return -1 ; +return 0 ; }
include/linux/nls.h +116 -0
147 lines not on this page · raw message
Signature · 1 line
2.19.0
PATCH RESEND v2 11/25 nls: ascii: Support validation and normalization operations
In reply to Gabriel Krisman Bertazivalidation is trivial. Any byte that has the MSB set is an invalid sequence.
Casefold can be implemented with uppercase or lowercase, and we have no specification on that. Callers should be safe using either of them, as long as it doesn't change.
--- fs/nls/nls_ascii.c | 50 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/nls.h | 8 ++++++++ 2 files changed, 58 insertions(+)
fs/nls/nls_ascii.c +50 -0
@@ -12,6 +12,7 @@#include <linux/string.h> #include <linux/nls.h> #include <linux/errno.h> +#include <linux/slab.h> static const wchar_t charset2uni[256 ] = {/* 0x00*/
@@ -117,6 +118,8 @@ static const unsigned char charset2upper[256] = { 0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, /* 0x78-0x7f */ }; +#define VALID_ASCII (c) (c < 128) +static int uni2char (wchar_t uni,unsigned char *out,int boundlen) {const unsigned char *uni2charset;
@@ -142,6 +145,16 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)return 1 ; } +static int ascii_validate (const struct nls_table *table, +const unsigned char *str,size_t len) +{ +int i; +for (i =0 ; i < len && str[i]; i++) +if (!VALID_ASCII (str[i])) +return -1 ; +return 0 ; +} +static unsigned char charset_tolower (const struct nls_table *table,unsigned int c){return charset2lower[c];
@@ -152,11 +165,36 @@ static unsigned char charset_toupper(const struct nls_table *table,return charset2upper[c]; } +static int ascii_casefold (const struct nls_table *charset, +const unsigned char *str,size_t len, +unsigned char *dest,size_t dlen) +{ +unsigned int i; + +if (dlen < len) +return -EINVAL ; + +for (i =0 ; i < len; i++) { +if (IS_STRICT_MODE (charset) && !VALID_ASCII (str[i])) +return -EINVAL ; + +if (IS_CASEFOLD_TYPE_ASCII_TOLOWER (charset)) + dest[i] =charset_tolower (charset, str[i]); +else + dest[i] =charset_toupper (charset, str[i]); + } + dest[len] ='\0' ; + +return len; +} +static const struct nls_ops charset_ops = { + .validate = ascii_validate, .lowercase = charset_toupper, .uppercase = charset_tolower, .uni2char = uni2char, .char2uni = char2uni, + .casefold = ascii_casefold, };static struct nls_charset nls_charset;
@@ -165,9 +203,21 @@ static struct nls_table table = { .ops = &charset_ops, }; +struct nls_table *ascii_load_table (const char *version,unsigned int flags) +{ +if (flags & ~(NLS_STRICT_MODE ) || + (flags &NLS_NORMALIZATION_TYPE_MASK ) !=NLS_NORMALIZATION_TYPE_PLAIN ) +return ERR_PTR (-EINVAL ); + + table.flags = flags; +return &table; +} + +static struct nls_charset nls_charset = { .charset ="ascii" , .tables = &table, + .load_table = ascii_load_table, };static int __init init_nls_ascii (void )
include/linux/nls.h +8 -0
@@ -178,6 +178,14 @@ IS_CASEFOLD_TYPE_##charset##_##type(const struct nls_table *c) \ NLS_NORMALIZATION_FUNCS(ALL, PLAIN, NLS_NORMALIZATION_TYPE_PLAIN)NLS_CASEFOLD_FUNCS (ALL ,TOUPPER ,NLS_CASEFOLD_TYPE_TOUPPER ) +/* ASCII */ + +#define NLS_ASCII_CASEFOLD_TOUPPER NLS_CASEFOLD_TYPE_TOUPPER +#define NLS_ASCII_CASEFOLD_TOLOWER NLS_CASEFOLD_TYPE(1) + +NLS_CASEFOLD_FUNCS(ASCII, TOUPPER, NLS_ASCII_CASEFOLD_TOUPPER) +NLS_CASEFOLD_FUNCS (ASCII ,TOLOWER ,NLS_ASCII_CASEFOLD_TOLOWER ) + /* nls_base.c */extern int __register_nls (struct nls_charset *,struct module *);extern int unregister_nls (struct nls_charset *);
Signature · 1 line
2.19.0
PATCH RESEND v2 12/25 nls: utf8n: Add unicode character database files
In reply to Gabriel Krisman BertaziFrom: Olaf Weber <[email protected]>
Add files from the Unicode Character Database, version 10.0.0, to the source. A helper program that generates a trie used for normalization from these files is part of a separate commit.
- Notes on the update from 8.0.0 and 10.0.0:
The structure of ucd files and special cases have not experienced any changes between versions 8.0.0 and 10.0.0. 8.0.0 saw the addition of Cherokee LC characters, which is an interesting case for case-folding. The update is accompanied by new tests on the test_ucd module to catch specific cases. No changes to mkutf8data script was required for the update.
The actual files are not part of the commit submitted to the list because they are to big and would bounce. Still, they can be obtained by the following script:
FILES="CaseFolding.txt DerivedAge.txt extracted/DerivedCombiningClass.txt
DerivedCoreProperties.txt NormalizationCorrections.txt NormalizationTest.txt UnicodeData.txt"
VERSION=10.0.0 BASE=http://www.unicode.org/Public/${VERSION}/ucd
for i in ${FILES} ; do
wget "${BASE}/$i" -O fs/nls/ucd/$(basename ${i} .txt)-${VERSION}.txt
done
--- fs/nls/ucd/README | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 fs/nls/ucd/README
fs/nls/ucd/README new file +33 -0
@@ -0,0 +1,33 @@ +The files in this directory are part of the Unicode Character Database +for version 10.0.0 of the Unicode standard. + +The full set of files can be found here: + + http://www.unicode.org/Public/10.0.0/ucd/ + +The latest released version of the UCD can be found here: + + http://www.unicode.org/Public/UCD/latest/ + +The files in this directory are identical, except that they have been +renamed with a suffix indicating the unicode version. + +Individual source links: + + http://www.unicode.org/Public/10.0.0/ucd/CaseFolding.txt + http://www.unicode.org/Public/10.0.0/ucd/DerivedAge.txt + http://www.unicode.org/Public/10.0.0/ucd/extracted/DerivedCombiningClass.txt + http://www.unicode.org/Public/10.0.0/ucd/DerivedCoreProperties.txt + http://www.unicode.org/Public/10.0.0/ucd/NormalizationCorrections.txt + http://www.unicode.org/Public/10.0.0/ucd/NormalizationTest.txt + http://www.unicode.org/Public/10.0.0/ucd/UnicodeData.txt + +md5sums + + 7893b6e005c5a521319a0d12062ae122 CaseFolding-10.0.0.txt + a602e4b44de3350087e40f2eb2184898 DerivedAge-10.0.0.txt + 5abdeb21af4edcc5d1e4c0b5802fc7a7 DerivedCombiningClass-10.0.0.txt + eda11c2c2e3c308d9d3b90e2b3282024 DerivedCoreProperties-10.0.0.txt + 425ece5ffbecd0140d98c13ce05724aa NormalizationCorrections-10.0.0.txt + 7296fe7aa07d7d288e65d559af2ad49b NormalizationTest-10.0.0.txt + 2a52f30695dcc821f0f224650552beaf UnicodeData-10.0.0.txt
Signature · 1 line
2.19.0
PATCH RESEND v2 13/25 scripts: add trie generator for UTF-8
In reply to Gabriel Krisman BertaziFrom: Olaf Weber <[email protected]>
mkutf8data.c is the source for a program that generates utf8data.h, which contains the trie that utf8norm.c uses. The trie is generated from the Unicode 10.0.0 data files. The format of the utf8data[] table is described in utf8norm.c, which is added in the next patch.
--- fs/nls/Kconfig | 10 + fs/nls/Makefile | 13 + scripts/Makefile | 1 + scripts/mkutf8data.c | 3239 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 3263 insertions(+) create mode 100644 scripts/mkutf8data.c
fs/nls/Kconfig +10 -0
@@ -616,4 +616,14 @@ config NLS_UTF8 input/output character sets. Say Y here for the UTF-8 encoding of the Unicode/ISO9646 universal character set. +# +# utf8 normalization +# +config NLS_UTF8_NORMALIZATION + bool "UTF-8 normalization and casefolding support" + depends on NLS_UTF8 + help + Say Y here to enable utf8 NFKD normalization and casefolding + support. + endif # NLS
fs/nls/Makefile +13 -0
@@ -55,3 +55,16 @@ obj-$(CONFIG_NLS_MAC_INUIT) += mac-inuit.o obj-$(CONFIG_NLS_MAC_ROMANIAN) += mac-romanian.o obj-$(CONFIG_NLS_MAC_ROMAN) += mac-roman.o obj-$(CONFIG_NLS_MAC_TURKISH) += mac-turkish.o + +$(obj)/utf8data.h: $(srctree)/$(src)/ucd/*.txt $(objtree)/scripts/mkutf8data FORCE + $(call cmd,mkutf8data) +quiet_cmd_mkutf8data = MKUTF8DATA $@ + cmd_mkutf8data = $(objtree)/scripts/mkutf8data \ + -a $(srctree)/$(src)/ucd/DerivedAge-10.0.0.txt \ + -c $(srctree)/$(src)/ucd/DerivedCombiningClass-10.0.0.txt \ + -p $(srctree)/$(src)/ucd/DerivedCoreProperties-10.0.0.txt \ + -d $(srctree)/$(src)/ucd/UnicodeData-10.0.0.txt \ + -f $(srctree)/$(src)/ucd/CaseFolding-10.0.0.txt \ + -n $(srctree)/$(src)/ucd/NormalizationCorrections-10.0.0.txt \ + -t $(srctree)/$(src)/ucd/NormalizationTest-10.0.0.txt \ + -o $@
scripts/Makefile +1 -0
@@ -19,6 +19,7 @@ hostprogs-$(CONFIG_ASN1) += asn1_compiler hostprogs-$(CONFIG_MODULE_SIG) += sign-file hostprogs-$(CONFIG_SYSTEM_TRUSTED_KEYRING) += extract-cert hostprogs-$(CONFIG_SYSTEM_EXTRA_CERTIFICATE) += insert-sys-cert +hostprogs-$(CONFIG_NLS_UTF8_NORMALIZATION) += mkutf8data HOSTCFLAGS_sortextable.o = -I$(srctree)/tools/include HOSTCFLAGS_asn1_compiler.o = -I$(srctree)/include
scripts/mkutf8data.c new file +3239 -0
2,963 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.0
Part of the diff is not shown here. View the raw message.
PATCH RESEND v2 14/25 nls: utf8: Move nls-utf8{,-core}.c
In reply to Gabriel Krisman Bertazinls_utf8 will be generated from multiple files, so lets move the existing code to a -core suffix.
---
fs/nls/Makefile | 3 +++
fs/nls/{nls_utf8.c => nls_utf8-core.c} | 0
2 files changed, 3 insertions(+)
rename fs/nls/{nls_utf8.c => nls_utf8-core.c} (100%)fs/nls/Makefile +3 -0
@@ -43,7 +43,10 @@ obj-$(CONFIG_NLS_ISO8859_14) += nls_iso8859-14.o obj-$(CONFIG_NLS_ISO8859_15) += nls_iso8859-15.o obj-$(CONFIG_NLS_KOI8_R) += nls_koi8-r.o obj-$(CONFIG_NLS_KOI8_U) += nls_koi8-u.o nls_koi8-ru.o + obj-$(CONFIG_NLS_UTF8) += nls_utf8.o +nls_utf8-y += nls_utf8-core.o + obj-$(CONFIG_NLS_MAC_CELTIC) += mac-celtic.o obj-$(CONFIG_NLS_MAC_CENTEURO) += mac-centeuro.o obj-$(CONFIG_NLS_MAC_CROATIAN) += mac-croatian.o
fs/nls/nls_utf8-core.c renamed from fs/nls/nls_utf8.c +0 -0
Signature · 1 line
2.19.0
PATCH RESEND v2 15/25 nls: utf8: Introduce code for UTF-8 normalization
In reply to Gabriel Krisman BertaziFrom: Olaf Weber <[email protected]>
Supporting functions for UTF-8 normalization are in utf8norm.c with the header utf8norm.h. Two normalization forms are supported: nfkdi and nfkdicf.
nfkdi:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
nfkdicf:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
- Apply a full casefold (C + F).
For the purposes of the code, a string is valid UTF-8 if:
- The values encoded are 0x1..0x10FFFF.
- The surrogate codepoints 0xD800..0xDFFFF are not encoded.
- The shortest possible encoding is used for all values.
The supporting functions work on null-terminated strings (utf8 prefix) and on length-limited strings (utf8n prefix).
Quoted 1 line
From the original SGI patch and for conformity with coding standards,
the utf8data_t typedef was dropped, since it was just masking the struct keyword. On other occasions, namely utf8leaf_t and utf8trie_t, I decided to keep it, since they are simple pointers to memory buffers, and using uchars here wouldn't provide any more meaningful information.
Changes since RFC v2:
- Merge to NLS system
Changes since RFC v1:
- utf8_version_is_supported receives maj, min and rev as separate arguments. (Olaf Weber)
--- fs/nls/Makefile | 2 + fs/nls/nls_utf8-norm.c | 640 +++++++++++++++++++++++++++++++++++++++++ fs/nls/utf8n.h | 112 ++++++++ 3 files changed, 754 insertions(+) create mode 100644 fs/nls/nls_utf8-norm.c create mode 100644 fs/nls/utf8n.h
fs/nls/Makefile +2 -0
@@ -46,6 +46,7 @@ obj-$(CONFIG_NLS_KOI8_U) += nls_koi8-u.o nls_koi8-ru.o obj-$(CONFIG_NLS_UTF8) += nls_utf8.o nls_utf8-y += nls_utf8-core.o +nls_utf8-$(CONFIG_NLS_UTF8_NORMALIZATION) += nls_utf8-norm.o obj-$(CONFIG_NLS_MAC_CELTIC) += mac-celtic.o obj-$(CONFIG_NLS_MAC_CENTEURO) += mac-centeuro.o
@@ -59,6 +60,7 @@ obj-$(CONFIG_NLS_MAC_ROMANIAN) += mac-romanian.o obj-$(CONFIG_NLS_MAC_ROMAN) += mac-roman.o obj-$(CONFIG_NLS_MAC_TURKISH) += mac-turkish.o +$(obj)/nls_utf8-norm.o: $(obj)/utf8data.h $(obj)/utf8data.h: $(srctree)/$(src)/ucd/*.txt $(objtree)/scripts/mkutf8data FORCE $(call cmd,mkutf8data) quiet_cmd_mkutf8data = MKUTF8DATA $@
fs/nls/nls_utf8-norm.c new file +640 -0
640 lines not on this page · raw message
fs/nls/utf8n.h new file +112 -0
112 lines not on this page · raw message
Signature · 1 line
2.19.0
PATCH RESEND v2 16/25 nls: utf8n: reduce the size of utf8data[]
In reply to Gabriel Krisman BertaziFrom: Olaf Weber <[email protected]>
Remove the Hangul decompositions from the utf8data trie, and do algorithmic decomposition to calculate them on the fly. To store the decomposition the caller of utf8lookup()/utf8nlookup() must provide a 12-byte buffer, which is used to synthesize a leaf with the decomposition. Trie size is reduced from 245kB to 90kB.
[Extract robustness fixes and merge back to original mkutf8data.c patch]
--- fs/nls/nls_utf8-norm.c | 191 +++++++++++++++++++++++--- fs/nls/utf8n.h | 4 + scripts/mkutf8data.c | 295 ++++++++++++++++++++++++++++++++++++----- 3 files changed, 435 insertions(+), 55 deletions(-)
fs/nls/nls_utf8-norm.c +171 -20
@@ -98,6 +98,38 @@ static inline int utf8clen(const char *s)return 1 + (c >=0xC0 ) + (c >=0xE0 ) + (c >=0xF0 ); } +/* +* Decode a 3-byte UTF-8 sequence. +*/ +static unsigned int +utf8decode3 (const char *str) +{ +unsigned int uc; + + uc = *str++ &0x0F ; + uc <<=6 ; + uc |= *str++ &0x3F ; + uc <<=6 ; + uc |= *str++ &0x3F ; + +return uc; +} + +/* +* Encode a 3-byte UTF-8 sequence. +*/ +static int +utf8encode3 (char *str,unsigned int val) +{ + str[2 ] = (val &0x3F ) |0x80 ; + val >>=6 ; + str[1 ] = (val &0x3F ) |0x80 ; + val >>=6 ; + str[0 ] = val |0xE0 ; + +return 3 ; +} +/* * utf8trie_t *
@@ -159,7 +191,8 @@ typedef const unsigned char utf8trie_t;* characters with the Default_Ignorable_Code_Point property. * These do affect normalization, as they all have CCC 0. * -* The decompositions in the trie have been fully expanded. +* The decompositions in the trie have been fully expanded, with the +* exception of Hangul syllables, which are decomposed algorithmically. * * Casefolding, if applicable, is also done using decompositions. *
@@ -179,6 +212,105 @@ typedef const unsigned char utf8leaf_t;#define STOPPER (0)#define DECOMPOSE (255) +/* Marker for hangul syllable decomposition. */ +#define HANGUL ((char)(255)) +/* Size of the synthesized leaf used for Hangul syllable decomposition. */ +#define UTF8HANGULLEAF (12) + +/* +* Hangul decomposition (algorithm from Section 3.12 of Unicode 6.3.0) +* +* AC00;<Hangul Syllable, First>;Lo;0;L;;;;;N;;;;; +* D7A3;<Hangul Syllable, Last>;Lo;0;L;;;;;N;;;;; +* +* SBase = 0xAC00 +* LBase = 0x1100 +* VBase = 0x1161 +* TBase = 0x11A7 +* LCount = 19 +* VCount = 21 +* TCount = 28 +* NCount = 588 (VCount * TCount) +* SCount = 11172 (LCount * NCount) +* +* Decomposition: +* SIndex = s - SBase +* +* LV (Canonical/Full) +* LIndex = SIndex / NCount +* VIndex = (Sindex % NCount) / TCount +* LPart = LBase + LIndex +* VPart = VBase + VIndex +* +* LVT (Canonical) +* LVIndex = (SIndex / TCount) * TCount +* TIndex = (Sindex % TCount) +* LVPart = SBase + LVIndex +* TPart = TBase + TIndex +* +* LVT (Full) +* LIndex = SIndex / NCount +* VIndex = (Sindex % NCount) / TCount +* TIndex = (Sindex % TCount) +* LPart = LBase + LIndex +* VPart = VBase + VIndex +* if (TIndex == 0) { +* d = <LPart, VPart> +* } else { +* TPart = TBase + TIndex +* d = <LPart, TPart, VPart> +* } +*/ + +/* Constants */ +#define SB (0xAC00) +#define LB (0x1100) +#define VB (0x1161) +#define TB (0x11A7) +#define LC (19) +#define VC (21) +#define TC (28) +#define NC (VC * TC) +#define SC (LC * NC) + +/* Algorithmic decomposition of hangul syllable. */ +static utf8leaf_t * +utf8hangul (const char *str,unsigned char *hangul) +{ +unsigned int si; +unsigned int li; +unsigned int vi; +unsigned int ti; +unsigned char *h; + +/* Calculate the SI, LI, VI, and TI values. */ + si =utf8decode3 (str) -SB ; + li = si /NC ; + vi = (si %NC ) /TC ; + ti = si %TC ; + +/* Fill in base of leaf. */ + h = hangul; +LEAF_GEN (h) =2 ; +LEAF_CCC (h) =DECOMPOSE ; + h +=2 ; + +/* Add LPart, a 3-byte UTF-8 sequence. */ + h +=utf8encode3 ((char *)h, li +LB ); + +/* Add VPart, a 3-byte UTF-8 sequence. */ + h +=utf8encode3 ((char *)h, vi +VB ); + +/* Add TPart if required, also a 3-byte UTF-8 sequence. */ +if (ti) + h +=utf8encode3 ((char *)h, ti +TB ); + +/* Terminate string. */ + h[0 ] ='\0' ; + +return hangul; +} +/* * Use trie to scan s, touching at most len bytes. * Returns the leaf if one exists, NULL otherwise.
@@ -187,8 +319,8 @@ typedef const unsigned char utf8leaf_t;* is well-formed and corresponds to a known unicode code point. The * shorthand for this will be "is valid UTF-8 unicode". */ -static utf8leaf_t *utf8nlookup (const struct utf8data *data,const char *s, -size_t len) +static utf8leaf_t *utf8nlookup (const struct utf8data *data, +unsigned char *hangul,const char *s,size_t len) {utf8trie_t *trie = utf8data + data->offset;int offlen;
@@ -226,8 +358,7 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s, trie++; }else {/* No right node. */ - node =0 ; - trie =NULL ; +return NULL ; } }else {/* Left leg */
@@ -237,8 +368,7 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s, trie += offlen +1 ; }else if (*trie &RIGHTPATH ) {/* No left node. */ - node =0 ; - trie =NULL ; +return NULL ; }else {/* Left node after this node */ node = (*trie &TRIENODE );
@@ -246,6 +376,14 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s, } } } +/* +* Hangul decomposition is done algorithmically. These are the +* codepoints >= 0xAC00 and <= 0xD7A3. Their UTF-8 encoding is +* always 3 bytes long, so s has been advanced twice, and the +* start of the sequence is at s-2. +*/ +if (LEAF_CCC (trie) ==DECOMPOSE &&LEAF_STR (trie)[0 ] ==HANGUL ) + trie =utf8hangul (s -2 , hangul);return trie; }
@@ -255,9 +393,10 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s,* * Forwards to utf8nlookup(). */ -static utf8leaf_t *utf8lookup (const struct utf8data *data,const char *s) +static utf8leaf_t *utf8lookup (const struct utf8data *data, +unsigned char *hangul,const char *s) { -return utf8nlookup (data, s, (size_t )-1 ); +return utf8nlookup (data, hangul, s, (size_t )-1 ); }/*
@@ -270,11 +409,13 @@ int utf8agemax(const struct utf8data *data, const char *s)utf8leaf_t *leaf;int age =0 ;int leaf_age; +unsigned char hangul[UTF8HANGULLEAF ];if (!data)return -1 ; +while (*s) { - leaf =utf8lookup (data, s); + leaf =utf8lookup (data, hangul, s);if (!leaf)return -1 ;
@@ -297,12 +438,13 @@ int utf8agemin(const struct utf8data *data, const char *s)utf8leaf_t *leaf;int age;int leaf_age; +unsigned char hangul[UTF8HANGULLEAF ];if (!data)return -1 ; age = data->maxage;while (*s) { - leaf =utf8lookup (data, s); + leaf =utf8lookup (data, hangul, s);if (!leaf)return -1 ; leaf_age = utf8agetab[LEAF_GEN (leaf)];
@@ -323,11 +465,13 @@ int utf8nagemax(const struct utf8data *data, const char *s, size_t len)utf8leaf_t *leaf;int age =0 ;int leaf_age; +unsigned char hangul[UTF8HANGULLEAF ];if (!data)return -1 ; +while (len && *s) { - leaf =utf8nlookup (data, s, len); + leaf =utf8nlookup (data, hangul, s, len);if (!leaf)return -1 ; leaf_age = utf8agetab[LEAF_GEN (leaf)];
@@ -349,12 +493,13 @@ int utf8nagemin(const struct utf8data *data, const char *s, size_t len)utf8leaf_t *leaf;int leaf_age;int age; +unsigned char hangul[UTF8HANGULLEAF ];if (!data)return -1 ; age = data->maxage;while (len && *s) { - leaf =utf8nlookup (data, s, len); + leaf =utf8nlookup (data, hangul, s, len);if (!leaf)return -1 ; leaf_age = utf8agetab[LEAF_GEN (leaf)];
@@ -377,11 +522,12 @@ ssize_t utf8len(const struct utf8data *data, const char *s) {utf8leaf_t *leaf;size_t ret =0 ; +unsigned char hangul[UTF8HANGULLEAF ];if (!data)return -1 ;while (*s) { - leaf =utf8lookup (data, s); + leaf =utf8lookup (data, hangul, s);if (!leaf)return -1 ;if (utf8agetab[LEAF_GEN (leaf)] > data->maxage)
@@ -404,11 +550,12 @@ ssize_t utf8nlen(const struct utf8data *data, const char *s, size_t len) {utf8leaf_t *leaf;size_t ret =0 ; +unsigned char hangul[UTF8HANGULLEAF ];if (!data)return -1 ;while (len && *s) { - leaf =utf8nlookup (data, s, len); + leaf =utf8nlookup (data, hangul, s, len);if (!leaf)return -1 ;if (utf8agetab[LEAF_GEN (leaf)] > data->maxage)
@@ -531,10 +678,12 @@ int utf8byte(struct utf8cursor *u8c) }/* Look up the data for the current character. */ -if (u8c->p) - leaf =utf8lookup (u8c->data, u8c->s); -else - leaf =utf8nlookup (u8c->data, u8c->s, u8c->len); +if (u8c->p) { + leaf =utf8lookup (u8c->data, u8c->hangul, u8c->s); + }else { + leaf =utf8nlookup (u8c->data, u8c->hangul, + u8c->s, u8c->len); + }/* No leaf found implies that the input is a binary blob. */ if (!leaf)
@@ -555,7 +704,9 @@ int utf8byte(struct utf8cursor *u8c) ccc =STOPPER ; goto ccc_mismatch; } - leaf =utf8lookup (u8c->data, u8c->s); + + leaf =utf8lookup (u8c->data, u8c->hangul, u8c->s); + ccc =LEAF_CCC (leaf); }/*
fs/nls/utf8n.h +4 -0
@@ -76,6 +76,9 @@ extern int utf8nagemin(const struct utf8data *data, const char *s, size_t len);extern ssize_t utf8len (const struct utf8data *data,const char *s);extern ssize_t utf8nlen (const struct utf8data *data,const char *s,size_t len); +/* Needed in struct utf8cursor below. */ +#define UTF8HANGULLEAF (12) +/* * Cursor structure used by the normalizer. */
@@ -89,6 +92,7 @@ struct utf8cursor {unsigned int slen;short int ccc;short int nccc; +unsigned char hangul[UTF8HANGULLEAF ]; };/*
scripts/mkutf8data.c +260 -35
592 lines not on this page · raw message
Signature · 1 line
2.19.0
PATCH RESEND v2 17/25 nls: utf8: Integrate utf8 normalization code with utf8 charset
In reply to Gabriel Krisman BertaziThis patch integrates the utf8n patches with the NLS utf8 charset by implementing the nls_ops operations and nls_charset table. The Normalization is done with NFKD, and Casefold is implemented using the NFKD+CF algorithm, implemented by Olaf Weber and SGI. The high level, strcmp, strncmp functions are implemented on top of the same utf8 code.
Utf-8 with normalization is exposed as optional on top of the existing utf8 charset, and disabled by default, to avoid changing the behavior of existing nls_utf8 users. To enable normalization, the specific normalization type must be set at load_table() time.
Changes since RFC v2:
- Integrate with NLS
- Merge utf8n with nls_utf8.
Changes since RFC v1:
- Change error return code from EIO to EINVAL. (Olaf Weber)
- Fix issues with strncmp/strcmp. (Olaf Weber)
- Remove stack buffer in normalization/casefold. (Olaf Weber)
- Include length parameter for second string on comparison functions.
- Change length type to size_t.
--- fs/nls/nls_utf8-core.c | 274 ++++++++++++++++++++++++++++++++++++++--- fs/nls/nls_utf8-norm.c | 6 + fs/nls/utf8n.h | 1 + include/linux/nls.h | 8 ++ 4 files changed, 275 insertions(+), 14 deletions(-)
fs/nls/nls_utf8-core.c +260 -14
@@ -6,10 +6,15 @@#include <linux/module.h> #include <linux/kernel.h> #include <linux/string.h> +#include <linux/slab.h> +#include <linux/parser.h> #include <linux/nls.h> #include <linux/errno.h> +#include "utf8n.h" +static unsigned char identity[256 ]; +static struct nls_charset utf8_info;static int uni2char (wchar_t uni,unsigned char *out,int boundlen) {
@@ -50,22 +55,262 @@ static unsigned char charset_toupper(const struct nls_table *table,return identity[c]; } -static const struct nls_ops charset_ops = { - .lowercase = charset_toupper, - .uppercase = charset_tolower, - .uni2char = uni2char, - .char2uni = char2uni, -}; +#ifdef CONFIG_NLS_UTF8_NORMALIZATION + +static int utf8_validate (const struct nls_table *charset, +const unsigned char *str,size_t len) +{ +const struct utf8data *data =utf8nfkdi (charset->version); + +if (utf8nlen (data, str, len) <0 ) +return -1 ; +return 0 ; +} + +static int utf8_strncmp (const struct nls_table *charset, +const unsigned char *str1,size_t len1, +const unsigned char *str2,size_t len2) +{ +const struct utf8data *data =utf8nfkdi (charset->version); +struct utf8cursor cur1, cur2; +int c1, c2; + +if (utf8ncursor (&cur1, data, str1, len1) <0 ) + goto invalid_seq; + +if (utf8ncursor (&cur2, data, str2, len2) <0 ) + goto invalid_seq; + +do { + c1 =utf8byte (&cur1); + c2 =utf8byte (&cur2); + +if (c1 <0 || c2 <0 ) + goto invalid_seq; +if (c1 != c2) +return 1 ; + }while (c1); + +return 0 ; + +invalid_seq: +if (IS_STRICT_MODE (charset)) +return -EINVAL ; + +/* Treat the sequence as a binary blob. */ +if (len1 != len2) +return 1 ; + +return !!memcmp (str1, str2, len1); +} + +static int utf8_strncasecmp (const struct nls_table *charset, +const unsigned char *str1,size_t len1, +const unsigned char *str2,size_t len2) +{ +const struct utf8data *data =utf8nfkdicf (charset->version); +struct utf8cursor cur1, cur2; +int c1, c2; + +if (utf8ncursor (&cur1, data, str1, len1) <0 ) + goto invalid_seq; + +if (utf8ncursor (&cur2, data, str2, len2) <0 ) + goto invalid_seq; + +do { + c1 =utf8byte (&cur1); + c2 =utf8byte (&cur2); + +if (c1 <0 || c2 <0 ) + goto invalid_seq; +if (c1 != c2) +return 1 ; + }while (c1); + +return 0 ; + +invalid_seq: +if (IS_STRICT_MODE (charset)) +return -EINVAL ; + +/* Treat the sequence as a binary blob. */ +if (len1 != len2) +return 1 ; + +return !!memcmp (str1, str2, len1); +} + +static int utf8_casefold_nfkdcf (const struct nls_table *charset, +const unsigned char *str,size_t len, +unsigned char *dest,size_t dlen) +{ +const struct utf8data *data =utf8nfkdicf (charset->version); +struct utf8cursor cur; +size_t nlen =0 ; + +if (utf8ncursor (&cur, data, str, len) <0 ) + goto invalid_seq; + +for (nlen =0 ; nlen < dlen; nlen++) { + dest[nlen] =utf8byte (&cur); +if (!dest[nlen]) +return nlen; +if (dest[nlen] ==-1 ) +break ; + } + +invalid_seq: +if (IS_STRICT_MODE (charset)) +return -EINVAL ; + +/* Treat the sequence as a binary blob. */ +memcpy (dest, str, len); +return len; +} + +static int utf8_normalize_nfkd (const struct nls_table *charset, +const unsigned char *str, +size_t len,unsigned char *dest,size_t dlen) +{ +const struct utf8data *data =utf8nfkdi (charset->version); +struct utf8cursor cur; +ssize_t nlen =0 ; -static struct nls_charset nls_charset; -static struct nls_table table = { - .charset = &nls_charset, - .ops = &charset_ops, +if (utf8ncursor (&cur, data, str, len) <0 ) + goto invalid_seq; + +for (nlen =0 ; nlen < dlen; nlen++) { + dest[nlen] =utf8byte (&cur); +if (!dest[nlen]) +return nlen; +if (dest[nlen] ==-1 ) +break ; + } + +invalid_seq: +if (IS_STRICT_MODE (charset)) +return -EINVAL ; + +/* Treat the sequence as a binary blob. */ +memcpy (dest, str, len); +return len; +} + +static int utf8_parse_version (const char *version,unsigned int *maj, +unsigned int *min,unsigned int *rev) +{ +substring_t args[3 ]; +char *tmp; +const struct match_token token[] = { + {1 ,"%d.%d.%d" }, + {0 ,NULL } + }; +int ret =0 ; + + tmp =kstrdup (version,GFP_KERNEL ); +if (match_token (tmp, token, args) !=1 ) { + ret = -EINVAL ; + goto out; + } + +if (match_int (&args[0 ], maj) ||match_int (&args[1 ], min) || +match_int (&args[2 ], rev)) { + ret = -EINVAL ; + goto out; + } +out: +kfree (tmp); +return ret; +} +#endif + +struct utf8_table { +struct nls_table tbl; +struct nls_ops ops; }; -static struct nls_charset nls_charset = { +static void utf8_set_ops (struct utf8_table *utbl) +{ + utbl->ops.lowercase = charset_toupper; + utbl->ops.uppercase = charset_tolower; + utbl->ops.uni2char = uni2char; + utbl->ops.char2uni = char2uni; + +#ifdef CONFIG_NLS_UTF8_NORMALIZATION + utbl->ops.validate = utf8_validate; + +if (IS_NORMALIZATION_TYPE_UTF8_NFKD (&utbl->tbl)) { + utbl->ops.normalize = utf8_normalize_nfkd; + utbl->ops.strncmp = utf8_strncmp; + } + +if (IS_CASEFOLD_TYPE_UTF8_NFKDCF (&utbl->tbl)) { + utbl->ops.casefold = utf8_casefold_nfkdcf; + utbl->ops.strncasecmp = utf8_strncasecmp; + } +#endif + + utbl->tbl.ops = &utbl->ops; +} + +static struct nls_table *utf8_load_table (const char *version,unsigned int flags) +{ +struct utf8_table *utbl =NULL ; +unsigned int nls_version; + +#ifdef CONFIG_NLS_UTF8_NORMALIZATION +if (version) { +unsigned int maj, min, rev; + +if (utf8_parse_version (version, &maj, &min, &rev) <0 ) +return ERR_PTR (-EINVAL ); + +if (!utf8version_is_supported (maj, min, rev)) +return ERR_PTR (-EINVAL ); + + nls_version =UNICODE_AGE (maj, min, rev); + }else { + nls_version =utf8version_latest (); +printk (KERN_WARNING "UTF-8 version not specified. " +"Assuming latest supported version (%d.%d.%d)." , + (nls_version >>16 ) &0xff , (nls_version >>8 ) &0xff , + (nls_version &0xff )); + } +#else + nls_version =0 ; +#endif + + utbl =kzalloc (sizeof (struct utf8_table ),GFP_KERNEL ); +if (!utbl) +return ERR_PTR (-ENOMEM ); + + utbl->tbl.charset = &utf8_info; + utbl->tbl.version = nls_version; + utbl->tbl.flags = flags; +utf8_set_ops (utbl); + + utbl->tbl.next = utf8_info.tables; + utf8_info.tables = &utbl->tbl; + +return &utbl->tbl; +} + +static void utf8_cleanup_tables (void ) +{ +struct nls_table *tmp, *tbl = utf8_info.tables; + +while (tbl) { + tmp = tbl; + tbl = tbl->next; +kfree (tmp); + } + utf8_info.tables =NULL ; +} + +static struct nls_charset utf8_info = { .charset ="utf8" , - .tables = &table, + .load_table = utf8_load_table, };static int __init init_nls_utf8 (void )
@@ -74,12 +319,13 @@ static int __init init_nls_utf8(void)for (i=0 ; i<256 ; i++) identity[i] = i; -return register_nls (&nls_charset); +return register_nls (&utf8_info); } static void __exit exit_nls_utf8(void) { -unregister_nls (&nls_charset); +unregister_nls (&utf8_info); +utf8_cleanup_tables (); }module_init (init_nls_utf8)
fs/nls/nls_utf8-norm.c +6 -0
@@ -38,6 +38,12 @@ int utf8version_is_supported(u8 maj, u8 min, u8 rev) }EXPORT_SYMBOL (utf8version_is_supported); +int utf8version_latest () +{ +return utf8vers; +} +EXPORT_SYMBOL (utf8version_latest); +/* * UTF-8 valid ranges. *
fs/nls/utf8n.h +1 -0
@@ -32,6 +32,7 @@/* Highest unicode version supported by the data tables. */ extern int utf8version_is_supported (u8 maj,u8 min,u8 rev); +extern int utf8version_latest (void );/* * Look for the correct const struct utf8data for a unicode version.
include/linux/nls.h +8 -0
@@ -186,6 +186,14 @@ NLS_CASEFOLD_FUNCS(ALL, TOUPPER, NLS_CASEFOLD_TYPE_TOUPPER) NLS_CASEFOLD_FUNCS(ASCII, TOUPPER, NLS_ASCII_CASEFOLD_TOUPPER)NLS_CASEFOLD_FUNCS (ASCII ,TOLOWER ,NLS_ASCII_CASEFOLD_TOLOWER ) +/* UTF-8 */ + +#define NLS_UTF8_NORMALIZATION_TYPE_NFKD NLS_NORMALIZATION_TYPE(1) +#define NLS_UTF8_CASEFOLD_TYPE_NFKDCF NLS_CASEFOLD_TYPE(1) + +NLS_NORMALIZATION_FUNCS(UTF8, NFKD, NLS_UTF8_NORMALIZATION_TYPE_NFKD) +NLS_CASEFOLD_FUNCS (UTF8 ,NFKDCF ,NLS_UTF8_CASEFOLD_TYPE_NFKDCF ) + /* nls_base.c */extern int __register_nls (struct nls_charset *,struct module *);extern int unregister_nls (struct nls_charset *);
Signature · 1 line
2.19.0
PATCH RESEND v2 18/25 nls: utf8: Introduce test module for normalized utf8 implementation
In reply to Gabriel Krisman BertaziThis implements a in-kernel sanity test module for the utf8 normalization core. At probe time, it will run basic sequences through the utf8n core, to identify problems will equivalent sequences and normalization/casefold code. This is supposed to be useful for regression testing when adding support for a new version of utf8 to linux.
Changes since RFC v2:
- Merge with NLS
Changes since RFC v1:
- Include comparison tests for matching strings with different lengths.
- Include tests for characters included in unicode 8.0.0, 9.0.0 and 10.0.0.
--- fs/nls/Kconfig | 5 + fs/nls/Makefile | 1 + fs/nls/nls_utf8-selftest.c | 307 +++++++++++++++++++++++++++++++++++++ 3 files changed, 313 insertions(+) create mode 100644 fs/nls/nls_utf8-selftest.c
fs/nls/Kconfig +5 -0
@@ -626,4 +626,9 @@ config NLS_UTF8_NORMALIZATION Say Y here to enable utf8 NFKD normalization and casefolding support. +config NLS_UTF8_NORMALIZATION_SELFTEST + tristate "Test UTF-8 normalization support" + depends on NLS_UTF8 + default n + endif # NLS
fs/nls/Makefile +1 -0
@@ -47,6 +47,7 @@ obj-$(CONFIG_NLS_KOI8_U) += nls_koi8-u.o nls_koi8-ru.o obj-$(CONFIG_NLS_UTF8) += nls_utf8.o nls_utf8-y += nls_utf8-core.o nls_utf8-$(CONFIG_NLS_UTF8_NORMALIZATION) += nls_utf8-norm.o +obj-$(CONFIG_NLS_UTF8_NORMALIZATION_SELFTEST) += nls_utf8-selftest.o obj-$(CONFIG_NLS_MAC_CELTIC) += mac-celtic.o obj-$(CONFIG_NLS_MAC_CENTEURO) += mac-centeuro.o
fs/nls/nls_utf8-selftest.c new file +307 -0
307 lines not on this page · raw message
Signature · 1 line
2.19.0
PATCH RESEND v2 19/25 vfs: Handle case-exact lookup in d_add_ci
In reply to Gabriel Krisman BertaziThis prevents a soft hang if called d_add_ci is called from the FS layer, when doing a CI search but the result dentry is the exact match.
--- fs/dcache.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-)
fs/dcache.c +18 -15
@@ -2059,6 +2059,20 @@ struct dentry *d_obtain_root(struct inode *inode) }EXPORT_SYMBOL (d_obtain_root); +static inline bool d_same_name (const struct dentry *dentry, +const struct dentry *parent, +const struct qstr *name) +{ +if (likely (!(parent->d_flags &DCACHE_OP_COMPARE ))) { +if (dentry->d_name.len != name->len) +return false; +return dentry_cmp (dentry, name->name, name->len) ==0 ; + } +return parent->d_op->d_compare (dentry, + dentry->d_name.len, dentry->d_name.name, + name) ==0 ; +} +/** * d_add_ci - lookup or allocate new dentry with case-exact name * @inode: the inode case-insensitive lookup has found
@@ -2080,6 +2094,10 @@ struct dentry *d_add_ci(struct dentry *dentry, struct inode *inode, {struct dentry *found, *res; +/* Trivial case: CI search is exact match. */ +if (d_same_name (dentry, dentry->d_parent, name)) +return d_splice_alias (inode, dentry); +/* * First check if a dentry matching the name already exists, * if not go ahead and create it now.
@@ -2112,21 +2130,6 @@ struct dentry *d_add_ci(struct dentry *dentry, struct inode *inode, }EXPORT_SYMBOL (d_add_ci); - -static inline bool d_same_name (const struct dentry *dentry, -const struct dentry *parent, -const struct qstr *name) -{ -if (likely (!(parent->d_flags &DCACHE_OP_COMPARE ))) { -if (dentry->d_name.len != name->len) -return false; -return dentry_cmp (dentry, name->name, name->len) ==0 ; - } -return parent->d_op->d_compare (dentry, - dentry->d_name.len, dentry->d_name.name, - name) ==0 ; -} -/** * __d_lookup_rcu - search for a dentry (racy, store-free) * @parent: parent dentry
Signature · 1 line
2.19.0
Theodore Y. Ts'o
maintainerRe: [PATCH RESEND v2 19/25] vfs: Handle case-exact lookup in d_add_ci
Quoted 2 lines · Gabriel Krisman Bertazi
This prevents a soft hang if called
d_add_ciis called from the FS layer, when doing a CI search but the result dentry is the exact match.
This isn't the right way to fix this problem. Take a look at how xfs handles this in fs/xfs/xfs_iops.c:xfs_vn_ci_lookup(). This logic should be in the file system, not in d_add_ci(). Also, we don't want to use d_same_name(), since that is *not* guaranteed to do an exact match. It happens to do so for ext4 since we don't provide d_compare, but it's better just check for an exact match and call d_splice_alias() instead of d_add_ci() in ext4_lookup().
Also note that d_same_name() is *not* guaranteeed to do an exact match, in particular if the file system provides d_compare (which granted, ext4 doesn't right now). It's simpler to just do a direct strcmp in ext4_lookup.
- Ted
P.S. Apologies for not having a chance to look at this series in detail until now. It's been a crazy month...
Re: [PATCH RESEND v2 19/25] vfs: Handle case-exact lookup in d_add_ci
Quoted 17 lines · Theodore Y. Ts'o
Quoted 2 lines · Gabriel Krisman Bertazi
This prevents a soft hang if called
d_add_ciis called from the FS layer, when doing a CI search but the result dentry is the exact match.This isn't the right way to fix this problem. Take a look at how xfs handles this in fs/xfs/xfs_iops.c:xfs_vn_ci_lookup(). This logic should be in the file system, not in
d_add_ci(). Also, we don't want to used_same_name(), since that is *not* guaranteed to do an exact match. It happens to do so for ext4 since we don't provided_compare, but it's better just check for an exact match and calld_splice_alias()instead ofd_add_ci()inext4_lookup().Also note that
d_same_name()is *not* guaranteeed to do an exact match, in particular if the file system providesd_compare(which granted, ext4 doesn't right now). It's simpler to just do a direct strcmp inext4_lookup.
Quoted 4 lines
- Ted
P.S. Apologies for not having a chance to look at this series in detail until now. It's been a crazy month...
Hey Ted,
No worries. Thanks for the review.
I'm aware of how xfs does it. I was looking for a generic way but I see its problems now. I will have it addressed in the next iteration.
thanks.
Signature · 1 line
Gabriel Krisman Bertazi
PATCH RESEND v2 20/25 ext4: Include encoding information in the superblock
In reply to Gabriel Krisman BertaziSupport for encoding is considered an incompatible feature, since it has potential to create collisions of file names in existing filesystems. If the feature flag is not enabled, the entire filesystem will operate on opaque byte sequences, respecting the original behavior.
The charset data is encoded in a new field in the superblock using a magic number specific to ext4. This is the easiest way I found to avoid writing the name of the charset in the superblock. The magic number is mapped to the exact NLS table, but the mapping is specific to ext4. Since we don't have any commitment to support old encodings, the only encodings I am supporting right now is utf8n-10.0.0 and ascii, both using the NLS abstraction.
The current implementation prevents the user from enabling encoding and per-directory encryption on the same filesystem at the same time. The incompatibility between these features lies in how we do efficient directory searches when we cannot be sure the encryption of the user provided fname will match the actual hash stored in the disk without decrypting every directory entry, because of normalization cases. My quickest solution is to simply block the concurrent use of these features for now, and enable it later, once we have a better solution.
Changes since v1:
- Guard code with CONFIG_NLS.
- Use 16 bits for s_ioencoding.
- Split mount option from this patch
--- fs/ext4/ext4.h | 19 ++++++++++++- fs/ext4/super.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-)
fs/ext4/ext4.h +18 -1
@@ -1190,6 +1190,11 @@ extern void ext4_set_bits(void *bm, int cur, int len);/* Metadata checksum algorithm codes */ #define EXT4_CRC32C_CHKSUM 1 +/* Be careful when modifying these flags. The lower byte must match the +* NLS flags. */ + +#define EXT4_ENC_NLS_FL_MASK 0x00FF +/* * Structure of the super block */
@@ -1316,7 +1321,9 @@ struct ext4_super_block {__u8 s_first_error_time_hi;__u8 s_last_error_time_hi;__u8 s_pad[2 ]; -__le32 s_reserved[96 ];/* Padding to the end of the block */ +__le16 s_ioencoding;/* charset encoding */ +__le16 s_ioencoding_flags;/* charset encoding flags */ +__le32 s_reserved[95 ];/* Padding to the end of the block */ __le32 s_checksum;/* crc32c(superblock) */ };
@@ -1341,6 +1348,8 @@ struct ext4_super_block {/* Number of quota types we support */ #define EXT4_MAXQUOTAS 3 +struct ext4_sb_encodings ; +/* * fourth extended-fs super-block data in memory */
@@ -1390,6 +1399,11 @@ struct ext4_sb_info {struct kobject s_kobj;struct completion s_kobj_unregister;struct super_block *s_sb; +#ifdef CONFIG_NLS +const struct ext4_sb_encodings *encoding_info; +struct nls_table *encoding; +__u16 encoding_flags; +#endif /* Journaling */ struct journal_s *s_journal;
@@ -1665,6 +1679,7 @@ static inline void ext4_clear_state_flags(struct ext4_inode_info *ei)#define EXT4_FEATURE_INCOMPAT_LARGEDIR 0x4000/* >2GB or 3-lvl htree */ #define EXT4_FEATURE_INCOMPAT_INLINE_DATA 0x8000/* data in inode */ #define EXT4_FEATURE_INCOMPAT_ENCRYPT 0x10000 +#define EXT4_FEATURE_INCOMPAT_IOENCODING 0x20000#define EXT4_FEATURE_COMPAT_FUNCS (name, flagname) \ static inline bool ext4_has_feature_##name(struct super_block *sb) \
@@ -1753,6 +1768,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(csum_seed, CSUM_SEED) EXT4_FEATURE_INCOMPAT_FUNCS(largedir, LARGEDIR) EXT4_FEATURE_INCOMPAT_FUNCS(inline_data, INLINE_DATA) EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) +EXT4_FEATURE_INCOMPAT_FUNCS(ioencoding, IOENCODING)#define EXT2_FEATURE_COMPAT_SUPP EXT4_FEATURE_COMPAT_EXT_ATTR#define EXT2_FEATURE_INCOMPAT_SUPP (EXT4_FEATURE_INCOMPAT_FILETYPE| \
@@ -1780,6 +1796,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) EXT4_FEATURE_INCOMPAT_MMP | \ EXT4_FEATURE_INCOMPAT_INLINE_DATA | \ EXT4_FEATURE_INCOMPAT_ENCRYPT | \ + EXT4_FEATURE_INCOMPAT_IOENCODING | \ EXT4_FEATURE_INCOMPAT_CSUM_SEED | \ EXT4_FEATURE_INCOMPAT_LARGEDIR)#define EXT4_FEATURE_RO_COMPAT_SUPP (EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \
fs/ext4/super.c +73 -0
109 lines not on this page · raw message
Signature · 1 line
2.19.0
Re: [PATCH RESEND v2 20/25] ext4: Include encoding information in the superblock
Quoted 50 lines · Gabriel Krisman Bertazi
Support for encoding is considered an incompatible feature, since it has potential to create collisions of file names in existing filesystems. If the feature flag is not enabled, the entire filesystem will operate on opaque byte sequences, respecting the original behavior.
The charset data is encoded in a new field in the superblock using a magic number specific to ext4. This is the easiest way I found to avoid writing the name of the charset in the superblock. The magic number is mapped to the exact NLS table, but the mapping is specific to ext4. Since we don't have any commitment to support old encodings, the only encodings I am supporting right now is utf8n-10.0.0 and ascii, both using the NLS abstraction.
The current implementation prevents the user from enabling encoding and per-directory encryption on the same filesystem at the same time. The incompatibility between these features lies in how we do efficient directory searches when we cannot be sure the encryption of the user provided fname will match the actual hash stored in the disk without decrypting every directory entry, because of normalization cases. My quickest solution is to simply block the concurrent use of these features for now, and enable it later, once we have a better solution.
Changes since v1:
- Guard code withCONFIG_NLS.
- Use 16 bits fors_ioencoding.
- Split mount option from this patchSigned-off-by Gabriel Krisman Bertazi <[email protected]>--- fs/ext4/ext4.h | 19 ++++++++++++- fs/ext4/super.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-)fs/ext4/ext4.h +5
-0@@ -1190,6 +1190,11 @@ extern void ext4_set_bits(void *bm, int cur, int len);/* Metadata checksum algorithm codes */ #define EXT4_CRC32C_CHKSUM 1 +/* Be careful when modifying these flags. The lower byte must match the +* NLS flags. */ + +#define EXT4_ENC_NLS_FL_MASK 0x00FF +/* * Structure of the super block */
fs/ext4/ext4.h struct ext4_super_block { · line 1326
Quoted 7 lines
fs/ext4/ext4.h +3
-1@@ -1316,7 +1321,9 @@ struct ext4_super_block {__u8 s_first_error_time_hi;__u8 s_last_error_time_hi;__u8 s_pad[2 ]; -__le32 s_reserved[96 ];/* Padding to the end of the block */ +__le16 s_ioencoding;/* charset encoding */ +__le16 s_ioencoding_flags;/* charset encoding flags */ +__le32 s_reserved[95 ];/* Padding to the end of the block */
Please update Documentation/filesystems/ext4/super.rst with this when it appears after the merge window.
Quoted 25 lines
fs/ext4/ext4.h +7
-0__le32 s_checksum;/* crc32c(superblock) */ };@@ -1341,6 +1348,8 @@ struct ext4_super_block {/* Number of quota types we support */ #define EXT4_MAXQUOTAS 3 +struct ext4_sb_encodings ; +/* * fourth extended-fs super-block data in memory */ @@ -1390,6 +1399,11 @@ struct ext4_sb_info {struct kobject s_kobj;struct completion s_kobj_unregister;struct super_block *s_sb; +#ifdef CONFIG_NLS +const struct ext4_sb_encodings *encoding_info; +struct nls_table *encoding; +__u16 encoding_flags; +#endif /* Journaling */ struct journal_s *s_journal;
fs/ext4/ext4.h static inline void ext4_clear_state_flags(struct ext4_inode_info *ei) · line 1682
Quoted 4 lines
fs/ext4/ext4.h +1
-0@@ -1665,6 +1679,7 @@ static inline void ext4_clear_state_flags(struct ext4_inode_info *ei)#define EXT4_FEATURE_INCOMPAT_LARGEDIR 0x4000/* >2GB or 3-lvl htree */ #define EXT4_FEATURE_INCOMPAT_INLINE_DATA 0x8000/* data in inode */ #define EXT4_FEATURE_INCOMPAT_ENCRYPT 0x10000 +#define EXT4_FEATURE_INCOMPAT_IOENCODING 0x20000
/me wonders how the 'ioencoding' name came about? We're not encoding disk IO, just directory entries.
Quoted 141 lines
fs/ext4/ext4.h +2
-0#define EXT4_FEATURE_COMPAT_FUNCS (name, flagname) \ static inline bool ext4_has_feature_##name(struct super_block *sb) \@@ -1753,6 +1768,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(csum_seed, CSUM_SEED) EXT4_FEATURE_INCOMPAT_FUNCS(largedir, LARGEDIR) EXT4_FEATURE_INCOMPAT_FUNCS(inline_data, INLINE_DATA) EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) +EXT4_FEATURE_INCOMPAT_FUNCS(ioencoding, IOENCODING)#define EXT2_FEATURE_COMPAT_SUPP EXT4_FEATURE_COMPAT_EXT_ATTR#define EXT2_FEATURE_INCOMPAT_SUPP (EXT4_FEATURE_INCOMPAT_FILETYPE| \@@ -1780,6 +1796,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) EXT4_FEATURE_INCOMPAT_MMP | \ EXT4_FEATURE_INCOMPAT_INLINE_DATA | \ EXT4_FEATURE_INCOMPAT_ENCRYPT | \ + EXT4_FEATURE_INCOMPAT_IOENCODING | \ EXT4_FEATURE_INCOMPAT_CSUM_SEED | \ EXT4_FEATURE_INCOMPAT_LARGEDIR)#define EXT4_FEATURE_RO_COMPAT_SUPP (EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \fs/ext4/super.c +73
-0@@ -42,6 +42,7 @@#include <linux/cleancache.h> #include <linux/uaccess.h> #include <linux/iversion.h> +#include <linux/nls.h> #include <linux/kthread.h> #include <linux/freezer.h> @@ -1010,6 +1011,10 @@ static void ext4_put_super(struct super_block *sb)crypto_free_shash (sbi->s_chksum_driver);kfree (sbi->s_blockgroup_lock);fs_put_dax (sbi->s_daxdev); +#ifdef CONFIG_NLS +unload_nls (sbi->encoding); +kfree (sbi->encoding_info); +#endif kfree (sbi); }@@ -1703,6 +1708,31 @@ static const struct mount_opts { {Opt_err, 0, 0} }; +#ifdef CONFIG_NLS +static const struct ext4_sb_encodings { +char *name; +char *version; +} ext4_sb_encoding_map[] = { +/* 0x0 */ {"ascii" ,NULL }, +/* 0x1 */ {"utf8" ,"10.0.0" }, +}; + +static int +ext4_sb_read_encoding (const struct ext4_super_block *es, +const struct ext4_sb_encodings **encoding,__u16 *flags) +{ +__u16 magic =le16_to_cpu (es->s_ioencoding); + +if (magic >=ARRAY_SIZE (ext4_sb_encoding_map)) +return -EINVAL ; + + *encoding = &ext4_sb_encoding_map[magic]; + *flags =le16_to_cpu (es->s_ioencoding_flags); + +return 0 ; +} +#endif +static int handle_mount_opt (struct super_block *sb,char *opt,int token,substring_t *args,unsigned long *journal_devnum,unsigned int *journal_ioprio,int is_remount)@@ -3515,6 +3545,11 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)int err =0 ;unsigned int journal_ioprio =DEFAULT_JOURNAL_IOPRIO ;ext4_group_t first_not_zeroed; +#ifdef CONFIG_NLS +struct nls_table *encoding; +const struct ext4_sb_encodings *encoding_info; +__u16 nls_flags; +#endif if ((data && !orig_data) || !sbi) goto out_free_base;@@ -3687,6 +3722,39 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent) &journal_ioprio, 0)) goto failed_mount; +#ifdef CONFIG_NLS +if (ext4_has_feature_ioencoding (sb) && !sbi->encoding) { +if (ext4_has_feature_encrypt (sb)) { +ext4_msg (sb,KERN_ERR , +"Can't mount with both encoding and encryption" ); + goto failed_mount; + } + +if (ext4_sb_read_encoding (es, &encoding_info, &nls_flags)) { +ext4_msg (sb,KERN_ERR , +"Encoding requested by superblock is unknown" ); + goto failed_mount; + } + + encoding =load_nls_version (encoding_info->name, + encoding_info->version, + nls_flags &EXT4_ENC_NLS_FL_MASK ); +if (IS_ERR (encoding)) { +ext4_msg (sb,KERN_ERR ,"can't mount with superblock charset: " +"%s-%s not supported by the kernel. flags: 0x%x" , + encoding_info->name, encoding_info->version, + nls_flags); + goto failed_mount; + } +ext4_msg (sb,KERN_INFO ,"Using encoding defined by superblock: " +"%s-%s with flags 0x%hx" , encoding_info->name, + encoding_info->version?:"\b" , nls_flags); + + sbi->encoding = encoding; + sbi->encoding_flags = nls_flags; + } +#endif +if (test_opt (sb,DATA_FLAGS ) ==EXT4_MOUNT_JOURNAL_DATA ) {printk_once (KERN_WARNING "EXT4-fs: Warning: mounting " "with data=journal disables delayed "@@ -4527,6 +4595,11 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent) failed_mount:if (sbi->s_chksum_driver)crypto_free_shash (sbi->s_chksum_driver); + +#ifdef CONFIG_NLS +unload_nls (sbi->encoding); +#endif +#ifdef CONFIG_QUOTA for (i =0 ; i <EXT4_MAXQUOTAS ; i++)kfree (sbi->s_qf_names[i]);Signature · 1 line
2.19.0
Re: [PATCH RESEND v2 20/25] ext4: Include encoding information in the superblock
Quoted 53 lines · Darrick J. Wong
Quoted 52 lines · Gabriel Krisman Bertazi
Support for encoding is considered an incompatible feature, since it has potential to create collisions of file names in existing filesystems. If the feature flag is not enabled, the entire filesystem will operate on opaque byte sequences, respecting the original behavior.
The charset data is encoded in a new field in the superblock using a magic number specific to ext4. This is the easiest way I found to avoid writing the name of the charset in the superblock. The magic number is mapped to the exact NLS table, but the mapping is specific to ext4. Since we don't have any commitment to support old encodings, the only encodings I am supporting right now is utf8n-10.0.0 and ascii, both using the NLS abstraction.
The current implementation prevents the user from enabling encoding and per-directory encryption on the same filesystem at the same time. The incompatibility between these features lies in how we do efficient directory searches when we cannot be sure the encryption of the user provided fname will match the actual hash stored in the disk without decrypting every directory entry, because of normalization cases. My quickest solution is to simply block the concurrent use of these features for now, and enable it later, once we have a better solution.
Changes since v1:
- Guard code withCONFIG_NLS.
- Use 16 bits fors_ioencoding.
- Split mount option from this patchSigned-off-by Gabriel Krisman Bertazi <[email protected]>--- fs/ext4/ext4.h | 19 ++++++++++++- fs/ext4/super.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-)fs/ext4/ext4.h +5
-0@@ -1190,6 +1190,11 @@ extern void ext4_set_bits(void *bm, int cur, int len);/* Metadata checksum algorithm codes */ #define EXT4_CRC32C_CHKSUM 1 +/* Be careful when modifying these flags. The lower byte must match the +* NLS flags. */ + +#define EXT4_ENC_NLS_FL_MASK 0x00FF +/* * Structure of the super block */ @@ -1316,7 +1321,9 @@ struct ext4_super_block {__u8 s_first_error_time_hi;__u8 s_last_error_time_hi;
Quoted 8 lines
fs/ext4/ext4.h struct ext4_super_block { · line 1326
Quoted 5 lines
fs/ext4/ext4.h +3
-1__u8 s_pad[2 ]; -__le32 s_reserved[96 ];/* Padding to the end of the block */ +__le16 s_ioencoding;/* charset encoding */ +__le16 s_ioencoding_flags;/* charset encoding flags */ +__le32 s_reserved[95 ];/* Padding to the end of the block */ Please update
Documentation/filesystems/ext4/super.rstwith this when it appears after the merge window.
Thanks, will do!
Quoted 23 lines
Quoted 23 lines
fs/ext4/ext4.h +7
-0__le32 s_checksum;/* crc32c(superblock) */ };@@ -1341,6 +1348,8 @@ struct ext4_super_block {/* Number of quota types we support */ #define EXT4_MAXQUOTAS 3 +struct ext4_sb_encodings ; +/* * fourth extended-fs super-block data in memory */ @@ -1390,6 +1399,11 @@ struct ext4_sb_info {struct kobject s_kobj;struct completion s_kobj_unregister;struct super_block *s_sb; +#ifdef CONFIG_NLS +const struct ext4_sb_encodings *encoding_info; +struct nls_table *encoding; +__u16 encoding_flags; +#endif
Quoted 10 lines
fs/ext4/ext4.h static inline void ext4_clear_state_flags(struct ext4_inode_info *ei) · line 1682
Quoted 7 lines
fs/ext4/ext4.h +1
-0/* Journaling */ struct journal_s *s_journal;@@ -1665,6 +1679,7 @@ static inline void ext4_clear_state_flags(struct ext4_inode_info *ei)#define EXT4_FEATURE_INCOMPAT_LARGEDIR 0x4000/* >2GB or 3-lvl htree */ #define EXT4_FEATURE_INCOMPAT_INLINE_DATA 0x8000/* data in inode */ #define EXT4_FEATURE_INCOMPAT_ENCRYPT 0x10000 +#define EXT4_FEATURE_INCOMPAT_IOENCODING 0x20000/me wonders how the 'ioencoding' name came about? We're not encoding disk IO, just directory entries.
Right hehe. I think I based this name on the iocharset option from fat and didn't think twice about it. We definitely could have a better name.
Quoted 142 lines
Quoted 142 lines
fs/ext4/ext4.h +2
-0#define EXT4_FEATURE_COMPAT_FUNCS (name, flagname) \ static inline bool ext4_has_feature_##name(struct super_block *sb) \@@ -1753,6 +1768,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(csum_seed, CSUM_SEED) EXT4_FEATURE_INCOMPAT_FUNCS(largedir, LARGEDIR) EXT4_FEATURE_INCOMPAT_FUNCS(inline_data, INLINE_DATA) EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) +EXT4_FEATURE_INCOMPAT_FUNCS(ioencoding, IOENCODING)#define EXT2_FEATURE_COMPAT_SUPP EXT4_FEATURE_COMPAT_EXT_ATTR#define EXT2_FEATURE_INCOMPAT_SUPP (EXT4_FEATURE_INCOMPAT_FILETYPE| \@@ -1780,6 +1796,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) EXT4_FEATURE_INCOMPAT_MMP | \ EXT4_FEATURE_INCOMPAT_INLINE_DATA | \ EXT4_FEATURE_INCOMPAT_ENCRYPT | \ + EXT4_FEATURE_INCOMPAT_IOENCODING | \ EXT4_FEATURE_INCOMPAT_CSUM_SEED | \ EXT4_FEATURE_INCOMPAT_LARGEDIR)#define EXT4_FEATURE_RO_COMPAT_SUPP (EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \fs/ext4/super.c +73
-0@@ -42,6 +42,7 @@#include <linux/cleancache.h> #include <linux/uaccess.h> #include <linux/iversion.h> +#include <linux/nls.h> #include <linux/kthread.h> #include <linux/freezer.h> @@ -1010,6 +1011,10 @@ static void ext4_put_super(struct super_block *sb)crypto_free_shash (sbi->s_chksum_driver);kfree (sbi->s_blockgroup_lock);fs_put_dax (sbi->s_daxdev); +#ifdef CONFIG_NLS +unload_nls (sbi->encoding); +kfree (sbi->encoding_info); +#endif kfree (sbi); }@@ -1703,6 +1708,31 @@ static const struct mount_opts { {Opt_err, 0, 0} }; +#ifdef CONFIG_NLS +static const struct ext4_sb_encodings { +char *name; +char *version; +} ext4_sb_encoding_map[] = { +/* 0x0 */ {"ascii" ,NULL }, +/* 0x1 */ {"utf8" ,"10.0.0" }, +}; + +static int +ext4_sb_read_encoding (const struct ext4_super_block *es, +const struct ext4_sb_encodings **encoding,__u16 *flags) +{ +__u16 magic =le16_to_cpu (es->s_ioencoding); + +if (magic >=ARRAY_SIZE (ext4_sb_encoding_map)) +return -EINVAL ; + + *encoding = &ext4_sb_encoding_map[magic]; + *flags =le16_to_cpu (es->s_ioencoding_flags); + +return 0 ; +} +#endif +static int handle_mount_opt (struct super_block *sb,char *opt,int token,substring_t *args,unsigned long *journal_devnum,unsigned int *journal_ioprio,int is_remount)@@ -3515,6 +3545,11 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)int err =0 ;unsigned int journal_ioprio =DEFAULT_JOURNAL_IOPRIO ;ext4_group_t first_not_zeroed; +#ifdef CONFIG_NLS +struct nls_table *encoding; +const struct ext4_sb_encodings *encoding_info; +__u16 nls_flags; +#endif if ((data && !orig_data) || !sbi) goto out_free_base;@@ -3687,6 +3722,39 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent) &journal_ioprio, 0)) goto failed_mount; +#ifdef CONFIG_NLS +if (ext4_has_feature_ioencoding (sb) && !sbi->encoding) { +if (ext4_has_feature_encrypt (sb)) { +ext4_msg (sb,KERN_ERR , +"Can't mount with both encoding and encryption" ); + goto failed_mount; + } + +if (ext4_sb_read_encoding (es, &encoding_info, &nls_flags)) { +ext4_msg (sb,KERN_ERR , +"Encoding requested by superblock is unknown" ); + goto failed_mount; + } + + encoding =load_nls_version (encoding_info->name, + encoding_info->version, + nls_flags &EXT4_ENC_NLS_FL_MASK ); +if (IS_ERR (encoding)) { +ext4_msg (sb,KERN_ERR ,"can't mount with superblock charset: " +"%s-%s not supported by the kernel. flags: 0x%x" , + encoding_info->name, encoding_info->version, + nls_flags); + goto failed_mount; + } +ext4_msg (sb,KERN_INFO ,"Using encoding defined by superblock: " +"%s-%s with flags 0x%hx" , encoding_info->name, + encoding_info->version?:"\b" , nls_flags); + + sbi->encoding = encoding; + sbi->encoding_flags = nls_flags; + } +#endif +if (test_opt (sb,DATA_FLAGS ) ==EXT4_MOUNT_JOURNAL_DATA ) {printk_once (KERN_WARNING "EXT4-fs: Warning: mounting " "with data=journal disables delayed "@@ -4527,6 +4595,11 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent) failed_mount:if (sbi->s_chksum_driver)crypto_free_shash (sbi->s_chksum_driver); + +#ifdef CONFIG_NLS +unload_nls (sbi->encoding); +#endif +#ifdef CONFIG_QUOTA for (i =0 ; i <EXT4_MAXQUOTAS ; i++)kfree (sbi->s_qf_names[i]);Signature · 1 line
2.19.0
Signature · 1 line
Gabriel Krisman Bertazi
PATCH RESEND v2 21/25 ext4: Add encoding mount options
In reply to Gabriel Krisman BertaziThis patch implements two new mount options for ext4: encoding and encoding_flags.
The encoding option receives a string that identifies the NLS encoding to be used when mounting the filesystem. The user can optionally ask for a specific version by appending the version string after a dash.
The encoding_flags argument allows the user to specify how the NLS charset must behave. The exact behavior of the flags are defined at ext4.h.
encoding_flags is ignored if the user didn't provide an encoding.
--- fs/ext4/super.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+)
fs/ext4/super.c +91 -0
@@ -1409,6 +1409,7 @@ enum { Opt_dioread_nolock, Opt_dioread_lock, Opt_discard, Opt_nodiscard, Opt_init_itable, Opt_noinit_itable, Opt_max_dir_size_kb, Opt_nojournal_checksum, Opt_nombcache, + Opt_encoding, Opt_encoding_flags }; static const match_table_t tokens = {
@@ -1493,6 +1494,8 @@ static const match_table_t tokens = { {Opt_noinit_itable, "noinit_itable"}, {Opt_max_dir_size_kb, "max_dir_size_kb=%u"}, {Opt_test_dummy_encryption, "test_dummy_encryption"}, + {Opt_encoding, "encoding=%s"}, + {Opt_encoding_flags, "encoding_flags=%u"}, {Opt_nombcache, "nombcache"}, {Opt_nombcache, "no_mbcache"}, /* for backward compatibility */ {Opt_removed, "check=none"}, /* mount option from ext2/3 */
@@ -1705,6 +1708,8 @@ static const struct mount_opts { {Opt_max_dir_size_kb, 0, MOPT_GTE0}, {Opt_test_dummy_encryption, 0, MOPT_GTE0}, {Opt_nombcache, EXT4_MOUNT_NO_MBCACHE, MOPT_SET}, + {Opt_encoding, 0, MOPT_EXT4_ONLY | MOPT_STRING}, + {Opt_encoding_flags, 0, MOPT_EXT4_ONLY}, {Opt_err, 0, 0} };
@@ -1731,6 +1736,27 @@ ext4_sb_read_encoding(const struct ext4_super_block *es,return 0 ; } + +static struct ext4_sb_encodings *ext4_parse_encoding_opt (char *arg) +{ +char *split; +struct ext4_sb_encodings *info; +ssize_t len =strlen (arg) +1 ; + + info =kzalloc (sizeof (struct ext4_sb_encodings ) + len,GFP_KERNEL ); +if (!info) +return NULL ; + + info->name = ((char *) &info[1 ]); +strncpy (info->name, arg, len); + + split =strchr (info->name,'-' ); +if (split && *(split+1 )) { + *split ='\0' ; + info->version = split+1 ; + } +return info; +}#endif static int handle_mount_opt(struct super_block *sb, char *opt, int token,
@@ -1965,6 +1991,42 @@ static int handle_mount_opt(struct super_block *sb, char *opt, int token, sbi->s_mount_opt |= m->mount_opt; }else if (token == Opt_data_err_ignore) { sbi->s_mount_opt &= ~m->mount_opt; + }else if (token == Opt_encoding) { +int ret =-1 ; +#ifdef CONFIG_NLS +char *encoding =match_strdup (&args[0 ]); + +if (!encoding) +return -ENOMEM ; + +if (ext4_has_feature_encrypt (sb)) { +ext4_msg (sb,KERN_ERR , +"Can't mount with both encoding and encryption" ); + goto encoding_out; + } + + sbi->encoding_info =ext4_parse_encoding_opt (encoding); +if (!sbi->encoding_info) { +ext4_msg (sb,KERN_ERR , +"Encoding %s not supported by ext4" , encoding); + goto encoding_out; + } + + ret =0 ; +encoding_out: +kfree (encoding); +#else +ext4_msg (sb,KERN_INFO ,"encoding option not supported" ); +#endif +return ret; + }else if (token == Opt_encoding_flags) { +#ifdef CONFIG_NLS + sbi->encoding_flags = arg; +return 0 ; +#else +ext4_msg (sb,KERN_INFO ,"encoding flags option not supported" ); +return -1 ; +#endif }else {if (!args->from) arg =1 ;
@@ -2051,6 +2113,35 @@ static int parse_options(char *options, struct super_block *sb,return 0 ; } } + +#ifdef CONFIG_NLS +if (sbi->encoding_info) { + sbi->encoding =load_nls_version (sbi->encoding_info->name, + sbi->encoding_info->version, + sbi->encoding_flags); +if (IS_ERR (sbi->encoding)) { +ext4_msg (sb,KERN_ERR , +"Cannot load encoding: %s %s with flags 0x%hx" , + sbi->encoding_info->name, + sbi->encoding_info->version?:"\b" , + sbi->encoding_flags &EXT4_ENC_NLS_FL_MASK ); + + sbi->encoding =NULL ; + sbi->encoding_flags =0 ; +kfree (sbi->encoding_info); + +return 0 ; + } +ext4_msg (sb,KERN_INFO ,"Using encoding defined by mount option: " +"%s %s with flags 0x%hx" , sbi->encoding_info->name, + sbi->encoding_info->version?:"\b" , sbi->encoding_flags); + }else { +/* Make sure the flags are zeroed if the the user didn't +provide the encoding name. */ + sbi->encoding_flags =0 ; + } + +#endif return 1 ; }
Signature · 1 line
2.19.0
Theodore Y. Ts'o
maintainerRe: [PATCH RESEND v2 21/25] ext4: Add encoding mount options
Quoted 7 lines · Gabriel Krisman Bertazi
This patch implements two new mount options for ext4: encoding and
encoding_flags.The encoding option receives a string that identifies the NLS encoding to be used when mounting the filesystem. The user can optionally ask for a specific version by appending the version string after a dash.
Quoted 7 lines
The
encoding_flagsargument allows the user to specify how the NLS charset must behave. The exact behavior of the flags are defined atext4.h.
encoding_flagsis ignored if the user didn't provide an encoding.Signed-off-by Gabriel Krisman Bertazi <[email protected]>
It seems to me that adding support for setting the encoding parameters via mount options is a bad idea. The encoding is going to impact directory hash; which means if the file system has directories created using, say, ASCII as its encoding, and then the encoding changes to UTF-8, directory lookups won't work correctly. So I think this commit needs to be dropped, and support for setting the encoding needs to be added to e2fsprogs as the primary way encoding settings are made.
We need e2fsprogs support before this feature is ready for production use, since e2fsck needs to be able to properly rebuild directories.
Do you agree?
- Ted
Re: [PATCH RESEND v2 21/25] ext4: Add encoding mount options
Quoted 9 lines · Theodore Y. Ts'o
On Mon, Sep 24, 2018 at 05:56:51PM -0400, Gabriel Krisman Bertazi wrote:
It seems to me that adding support for setting the encoding parameters via mount options is a bad idea. The encoding is going to impact directory hash; which means if the file system has directories created using, say, ASCII as its encoding, and then the encoding changes to UTF-8, directory lookups won't work correctly. So I think this commit needs to be dropped, and support for setting the encoding needs to be added to e2fsprogs as the primary way encoding settings are made.
The main reason I had this patch is debugging, so I am ok with dropping it.
Quoted 4 lines
We need e2fsprogs support before this feature is ready for production use, since e2fsck needs to be able to properly rebuild directories.
Do you agree?
I have support for e2fsprogs in the branch I mentioned in the cover letter, I will rebase and start submitting it in the next iteration.
I am only supporting encoding selection at mkfs time, so I don't need to rebuild the hashes, (except for fsck errors). I'm not sure I care about changing the encoding after creation time, since this complicates things, for instance by increasing the risk of file name collisions. I'm also only allowing case-sensitiveness configuration changes on empty directories for the same reason.
I will start by submitting kernel/e2fsprogs patches to reserve the superblock bits. Do we agree on the current superblock format?
Also, what is the best list to send the NLS patches? fsdevel?
Signature · 1 line
Gabriel Krisman Bertazi
PATCH RESEND v2 22/25 ext4: Support encoding-aware file name lookups
In reply to Gabriel Krisman BertaziThis patch implements the actual support for encoding-aware file name lookups in ext4, based on the feature bit and the encoding stored in the superblock.
A filesystem that has the encoding feature set is able to find files even if the name used by userspace is not exact, but an equivalent string. This operation will be called and inexact-match name search.
Ext4 will only store exact-match names in the cache. Even if an inexact-match is issued, the exact-match name is what will be stored in the dcache. This is done to prevent unintentional duplications of dentries in the dcache, and is the same approach used by filesystems doing case-insensitive lookups already. We use d_add_ci(), to fix the names and prevent such duplication.
For now, negative lookups are not inserted in the dcache, since they would need to be invalidated anyway, because we can't trust missing file dentries. This is bad for performance but requires some leveraging of the vfs layer to fix. We can live without that for now.
DX is supported by modifying the hashes to make them encoding-aware. The new hashes are calculated as the hash of the normalized string, instead of the string directly. This allows us to efficiently search for file names without requiring the user to provide the exact name.
Changes since v1:
- Support normalized htree hashes.
- Guard code with CONFIG_NLS.
--- fs/ext4/ext4.h | 6 ++-- fs/ext4/hash.c | 34 +++++++++++++++++- fs/ext4/ialloc.c | 2 +- fs/ext4/inline.c | 2 +- fs/ext4/namei.c | 92 ++++++++++++++++++++++++++++++++++++++++++------ 5 files changed, 119 insertions(+), 17 deletions(-)
fs/ext4/ext4.h +3 -3
@@ -1192,7 +1192,7 @@ extern void ext4_set_bits(void *bm, int cur, int len);/* Be careful when modifying these flags. The lower byte must match the * NLS flags. */ - +#define EXT4_ENC_STRICT_MODE_FL 0x0001#define EXT4_ENC_NLS_FL_MASK 0x00FF/*
@@ -2396,8 +2396,8 @@ extern int ext4_check_all_de(struct inode *dir, struct buffer_head *bh,extern int ext4_sync_file (struct file *,loff_t ,loff_t ,int );/* hash.c */ -extern int ext4fs_dirhash (const char *name,int len,struct -dx_hash_info *hinfo); +extern int ext4fs_dirhash (const struct inode *dir,const char *name,int len, +struct dx_hash_info *hinfo);/* ialloc.c */ extern struct inode *__ext4_new_inode(handle_t *, struct inode *, umode_t,
fs/ext4/hash.c +33 -1
@@ -6,6 +6,7 @@*/ #include <linux/fs.h> +#include <linux/nls.h> #include <linux/compiler.h> #include <linux/bitops.h> #include "ext4.h"
@@ -196,7 +197,8 @@ static void str2hashbuf_unsigned(const char *msg, int len, __u32 *buf, int num)* represented, and whether or not the returned hash is 32 bits or 64 * bits. 32 bit hashes will return 0 for the minor hash. */ -int ext4fs_dirhash (const char *name,int len,struct dx_hash_info *hinfo) +static int __ext4fs_dirhash (const char *name,int len, +struct dx_hash_info *hinfo) {__u32 hash;__u32 minor_hash =0 ;
@@ -266,3 +268,33 @@ int ext4fs_dirhash(const char *name, int len, struct dx_hash_info *hinfo) hinfo->minor_hash = minor_hash;return 0 ; } + +int ext4fs_dirhash (const struct inode *dir,const char *name,int len, +struct dx_hash_info *hinfo) +{ +#ifdef CONFIG_NLS +const struct nls_table *charset =EXT4_SB (dir->i_sb)->encoding; +int r, dlen; +unsigned char *buff; + +if (len && charset) { + buff =kzalloc (sizeof (char ) *PATH_MAX ,GFP_KERNEL ); +if (!buff) +return -1 ; + + dlen =nls_normalize (charset, name, len, buff,PATH_MAX ); + +if (dlen <0 ) { +kfree (buff); + goto opaque_seq; + } + + r =__ext4fs_dirhash (buff, dlen, hinfo); + +kfree (buff); +return r; + } +opaque_seq: +#endif +return __ext4fs_dirhash (name, len, hinfo); +}
fs/ext4/ialloc.c +1 -1
@@ -455,7 +455,7 @@ static int find_group_orlov(struct super_block *sb, struct inode *parent,if (qstr) { hinfo.hash_version =DX_HASH_HALF_MD4 ; hinfo.seed = sbi->s_hash_seed; -ext4fs_dirhash (qstr->name, qstr->len, &hinfo); +ext4fs_dirhash (parent, qstr->name, qstr->len, &hinfo); grp = hinfo.hash; }else grp =prandom_u32 ();
fs/ext4/inline.c +1 -1
@@ -1404,7 +1404,7 @@ int htree_inlinedir_to_tree(struct file *dir_file, } } -ext4fs_dirhash (de->name, de->name_len, hinfo); +ext4fs_dirhash (dir, de->name, de->name_len, hinfo);if ((hinfo->hash < start_hash) || ((hinfo->hash == start_hash) && (hinfo->minor_hash < start_minor_hash)))
fs/ext4/namei.c +81 -11
201 lines not on this page · raw message
Signature · 1 line
2.19.0
PATCH RESEND v2 23/25 ext4: Implement encoding-aware dcache hooks
In reply to Gabriel Krisman BertaziThis patch implements dcache hooks to improve the handling of encoding-aware file names by the vfs dcache.
d_hash() is implemented as the hash of the normalized string, such that have a well-known bucket for all the equivalencies of the same string. d_compare uses the nls_strncmp() infrastructure, which should handle the comparison of equivalent names as well. If the filesystem's normalization type is PLAIN, though, we can just reuse the VFS hash.
Changes since v1:
- Use qstr->len instead of strlen.
- Guard code with CONFIG_NLS.
--- fs/ext4/dir.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ fs/ext4/ext4.h | 3 +++ fs/ext4/super.c | 6 ++++++ 3 files changed, 54 insertions(+)
fs/ext4/dir.c +45 -0
@@ -26,6 +26,7 @@#include <linux/buffer_head.h> #include <linux/slab.h> #include <linux/iversion.h> +#include <linux/nls.h> #include "ext4.h" #include "xattr.h"
@@ -662,3 +663,47 @@ const struct file_operations ext4_dir_operations = { .open = ext4_dir_open, .release = ext4_release_dir, }; + +#ifdef CONFIG_NLS +static int ext4_d_compare (const struct dentry *dentry,unsigned int len, +const char *str,const struct qstr *name) +{ +struct nls_table *charset =EXT4_SB (dentry->d_sb)->encoding; + +return nls_strncmp (charset, str, len, name->name, name->len); +} + +static int ext4_d_hash (const struct dentry *dentry,struct qstr *q) +{ +const struct nls_table *charset =EXT4_SB (dentry->d_sb)->encoding; +unsigned char *norm; +int len, ret =0 ; + +/* If normalization is TYPE_PLAIN, we can just reuse the vfs +* hash. */ +if (IS_NORMALIZATION_TYPE_ALL_PLAIN (charset)) +return 0 ; + + norm =kmalloc (PATH_MAX ,GFP_ATOMIC ); +if (!norm) +return -ENOMEM ; + + len =nls_normalize (charset, q->name, q->len, norm,PATH_MAX ); + +if (len <0 ) { + ret = -EINVAL ; + goto out; + } + + q->hash =full_name_hash (dentry, norm, len); + +out: +kfree (norm); +return ret; +} + +const struct dentry_operations ext4_dentry_ops = { + .d_hash = ext4_d_hash, + .d_compare = ext4_d_compare, +}; +#endif
fs/ext4/ext4.h +3 -0
@@ -2979,6 +2979,9 @@ static inline void ext4_unlock_group(struct super_block *sb,/* dir.c */ extern const struct file_operations ext4_dir_operations; +#ifdef CONFIG_NLS +extern const struct dentry_operations ext4_dentry_ops; +#endif /* file.c */ extern const struct inode_operations ext4_file_inode_operations;
fs/ext4/super.c +6 -0
@@ -4480,6 +4480,12 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)iput (root); goto failed_mount4; } + +#ifdef CONFIG_NLS +if (sbi->encoding) + sb->s_d_op = &ext4_dentry_ops; +#endif + sb->s_root =d_make_root (root);if (!sb->s_root) {ext4_msg (sb,KERN_ERR ,"get root dentry failed" );
Signature · 1 line
2.19.0
PATCH RESEND v2 24/25 ext4: Implement EXT4_CASEFOLD_FL flag
In reply to Gabriel Krisman BertaziCasefold is a flag applied to directories and inherited by its children which states that the directory requires case-insensitive searches. This flag can only be enabled on empty directories for filesystems that support the encoding feature, thus preventing collision of file names that only differ by case.
Enconding-awareness is also required because we consider the casefold operation not be defined for opaque byte sequences.
Changes since last version:
- Moved the CASEFOLD_FL to prevent collision with reserved Verity flag.
--- fs/ext4/dir.c | 30 ++++++++++++++++++++++-------- fs/ext4/ext4.h | 7 ++++--- fs/ext4/hash.c | 6 +++++- fs/ext4/inode.c | 4 +++- fs/ext4/ioctl.c | 18 ++++++++++++++++++ fs/ext4/namei.c | 13 ++++++++++--- include/linux/fs.h | 2 ++ 7 files changed, 64 insertions(+), 16 deletions(-)
fs/ext4/dir.c +22 -8
@@ -670,6 +670,10 @@ static int ext4_d_compare(const struct dentry *dentry, unsigned int len, {struct nls_table *charset =EXT4_SB (dentry->d_sb)->encoding; +if (IS_CASEFOLDED (dentry->d_parent->d_inode)) +return nls_strncasecmp (charset, str, len, name->name, + name->len); +return nls_strncmp (charset, str, len, name->name, name->len); }
@@ -679,16 +683,26 @@ static int ext4_d_hash(const struct dentry *dentry, struct qstr *q)unsigned char *norm;int len, ret =0 ; -/* If normalization is TYPE_PLAIN, we can just reuse the vfs -* hash. */ -if (IS_NORMALIZATION_TYPE_ALL_PLAIN (charset)) -return 0 ; +if (!IS_CASEFOLDED (dentry->d_inode)) { - norm =kmalloc (PATH_MAX ,GFP_ATOMIC ); -if (!norm) -return -ENOMEM ; +/* If normalization is TYPE_PLAIN, we can just reuse the +* VFS hash. +*/ +if (IS_NORMALIZATION_TYPE_ALL_PLAIN (charset)) +return 0 ; - len =nls_normalize (charset, q->name, q->len, norm,PATH_MAX ); + norm =kmalloc (PATH_MAX ,GFP_ATOMIC ); +if (!norm) +return -ENOMEM ; + + len =nls_normalize (charset, q->name, q->len, norm,PATH_MAX ); + }else { + norm =kmalloc (PATH_MAX ,GFP_ATOMIC ); +if (!norm) +return -ENOMEM ; + + len =nls_casefold (charset, q->name, q->len, norm,PATH_MAX ); + }if (len <0 ) { ret = -EINVAL ;
fs/ext4/ext4.h +4 -3
@@ -409,10 +409,11 @@ struct flex_groups {#define EXT4_EOFBLOCKS_FL 0x00400000/* Blocks allocated beyond EOF */ #define EXT4_INLINE_DATA_FL 0x10000000/* Inode has inline data. */ #define EXT4_PROJINHERIT_FL 0x20000000/* Create with parents projid */ +#define EXT4_CASEFOLD_FL 0x40000000/* Casefolded file */ #define EXT4_RESERVED_FL 0x80000000/* reserved for ext4 lib */ -#define EXT4_FL_USER_VISIBLE 0x304BDFFF/* User visible flags */ -#define EXT4_FL_USER_MODIFIABLE 0x204BC0FF/* User modifiable flags */ +#define EXT4_FL_USER_VISIBLE 0x704BDFFF/* User visible flags */ +#define EXT4_FL_USER_MODIFIABLE 0x604BC0FF/* User modifiable flags */ /* Flags we can manipulate with through EXT4_IOC_FSSETXATTR */ #define EXT4_FL_XFLAG_VISIBLE (EXT4_SYNC_FL | \
@@ -427,7 +428,7 @@ struct flex_groups { EXT4_SYNC_FL | EXT4_NODUMP_FL | EXT4_NOATIME_FL |\ EXT4_NOCOMPR_FL | EXT4_JOURNAL_DATA_FL |\ EXT4_NOTAIL_FL | EXT4_DIRSYNC_FL |\ - EXT4_PROJINHERIT_FL) + EXT4_PROJINHERIT_FL | EXT4_CASEFOLD_FL)/* Flags that are appropriate for regular files (all but dir-specific ones). */ #define EXT4_REG_FLMASK (~(EXT4_DIRSYNC_FL | EXT4_TOPDIR_FL))
fs/ext4/hash.c +5 -1
@@ -282,7 +282,11 @@ int ext4fs_dirhash(const struct inode *dir, const char *name, int len,if (!buff)return -1 ; - dlen =nls_normalize (charset, name, len, buff,PATH_MAX ); +if (!IS_CASEFOLDED (dir)) + dlen =nls_normalize (charset, name, len, buff, +PATH_MAX ); +else + dlen =nls_casefold (charset, name, len, buff,PATH_MAX );if (dlen <0 ) {kfree (buff);
fs/ext4/inode.c +3 -1
@@ -4713,9 +4713,11 @@ void ext4_set_inode_flags(struct inode *inode) new_fl |=S_DAX ;if (flags &EXT4_ENCRYPT_FL ) new_fl |=S_ENCRYPTED ; +if (flags &EXT4_CASEFOLD_FL ) + new_fl |=S_CASEFOLD ;inode_set_flags (inode, new_fl,S_SYNC |S_APPEND |S_IMMUTABLE |S_NOATIME |S_DIRSYNC |S_DAX | -S_ENCRYPTED ); +S_ENCRYPTED |S_CASEFOLD ); } static blkcnt_t ext4_inode_blocks(struct ext4_inode *raw_inode,
fs/ext4/ioctl.c +18 -0
@@ -210,6 +210,7 @@ static int ext4_ioctl_setflags(struct inode *inode,struct ext4_iloc iloc;unsigned int oldflags, mask, i;unsigned int jflag; +struct super_block *sb = inode->i_sb;/* Is it quota file? Do not allow user to mess with it */ if (ext4_is_quota_file (inode))
@@ -254,6 +255,23 @@ static int ext4_ioctl_setflags(struct inode *inode, goto flags_out; } +if ((flags ^ oldflags) &EXT4_CASEFOLD_FL ) { +if (!ext4_has_feature_ioencoding (sb)) { + err = -EOPNOTSUPP ; + goto flags_out; + } + +if (!S_ISDIR (inode->i_mode)) { + err = -ENOTDIR ; + goto flags_out; + } + +if (!ext4_empty_dir (inode)) { + err = -ENOTEMPTY ; + goto flags_out; + } + } + handle =ext4_journal_start (inode,EXT4_HT_INODE ,1 );if (IS_ERR (handle)) { err =PTR_ERR (handle);
fs/ext4/namei.c +10 -3
@@ -1277,9 +1277,16 @@ static inline bool ext4_match(const struct inode *parent,#ifdef CONFIG_NLS if (sbi->encoding) { -return !nls_strncmp (sbi->encoding, - de->name, de->name_len, - f.disk_name.name, f.disk_name.len); +if (!IS_CASEFOLDED (parent)) +return !nls_strncmp (sbi->encoding, + de->name, de->name_len, + fname->disk_name.name, + fname->disk_name.len); +else +return !nls_strncasecmp (sbi->encoding, + de->name, de->name_len, + fname->disk_name.name, + fname->disk_name.len); }#endif
include/linux/fs.h +2 -0
@@ -1877,6 +1877,7 @@ struct super_operations {#define S_DAX 0/* Make all the DAX code disappear */ #endif #define S_ENCRYPTED 16384/* Encrypted file (using fs/crypto/) */ +#define S_CASEFOLD 32768/* Casefolded file */ /* * Note that nosuid etc flags are inode-specific: setting some file-system
@@ -1917,6 +1918,7 @@ static inline bool sb_rdonly(const struct super_block *sb) { return sb->s_flags#define IS_NOSEC (inode) ((inode)->i_flags & S_NOSEC)#define IS_DAX (inode) ((inode)->i_flags & S_DAX)#define IS_ENCRYPTED (inode) ((inode)->i_flags & S_ENCRYPTED) +#define IS_CASEFOLDED (inode) ((inode)->i_flags & S_CASEFOLD)#define IS_WHITEOUT (inode) (S_ISCHR(inode->i_mode) && \ (inode)->i_rdev == WHITEOUT_DEV)
Signature · 1 line
2.19.0
PATCH RESEND v2 25/25 docs: ext4.rst: Document encoding and case-insensitive lookups
In reply to Gabriel Krisman BertaziIntroduces the encoding-awareness and case-insensitive features on ext4, explains some of the design decisions and the mount options to enabled it.
--- Documentation/filesystems/ext4/ext4.rst | 38 +++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
Documentation/filesystems/ext4/ext4.rst +38 -0
@@ -91,10 +91,39 @@ Currently Available * large block (up to pagesize) support * efficient new ordered mode in JBD2 and ext4 (avoid using buffer head to force the ordering) +* Encoding aware file names +* Case insensitive file name lookups [1] Filesystems with a block size of 1k may see a limit imposed by the directory hash tree having a maximum depth of two. +Encoding-aware file names and case-insensitive lookups +====================================================== + +Ext4 optionally supports filesystem-wide charset knowledge when handling +file names, which allows the user to perform file system lookups using +charset equivalent versions of the same file name, and optionally ensure +that no invalid names are held by the filesystem. charset encoding +awareness is also essential for performing case-insensitive lookups, +because it is what defines the casefold operation. + +The case-insensitive file name lookup feature is supported in a smaller +granularity, on a per-directory basis, allowing the user to mix +case-insensitive and case-sensitive directories in the same filesystem. +It is enabled by flipping a file attribute on an empty directory. For +the reason stated above, the filesystem must have encoding enabled to +use this feature. + +When we change from filenames as opaque byte sequences to seeing them as +encoded strings we need to address what happens when a program tries to +create a file with an invalid name. The Natural Language System within +the kernel leaves the decision of what to do in this case to the +filesystem, which select its preferred behavior by enabling/disabling +the strict mode in NLS. When Ext4 encounters one of those strings, it +falls back to considering the entire string as an opaque byte sequence, +which still allows the user to operate on that file but the +case-insensitive and equivalent sequence lookups won't work. + Options =======
@@ -363,6 +392,15 @@ i_version Enable 64-bit inode version support. This option is dax Use direct access (no page cache). See Documentation/filesystems/dax.txt. Note that this option is incompatible with data=journal. + +encoding Enable a specific encoding for file name lookups. + This cannot be used with per-directory encryption and + will fail on filesystems that have that flag enabled. + +encoding_flags A bitmask to configure how the encoding aware mechanism + should function. It specifies whether to refuse invalid + sequences and the specific normalization and casefold + operations to use. ======================= ======================================================= Data Mode
Signature · 1 line
2.19.0
Quoted 11 lines · Gabriel Krisman Bertazi
[Resend, but also rebased on top of Ted's dev branch.]
Hi Ted,
This patchset implements encoding support as a superblock feature, valid for the entire disk, and Case-insensitive lookups support as a per-directory feature, configured as an inode flag. Alongside the addition of the casefolding patch, which was not part of v1, I fixed some bugs and addressed the concerns raised regarding invalid sequences and what normalization we use.
Quoted 5 lines
My approach to solve the two problems above is to make things more flexible. An encoding flags field in the superblock registers what kind of normalization is used by the filesystem. Right now, the only encoding supported for utf8 is NFKD, for instance, but if we want to support a different normalization function in the future, we can be
Hmmm, I'm curious, why pick NFKD specifically? AFAICT Linux userspace environments (I only tried with GNOME and KDE) use NF[K]C. I saved a file with the name "café.txt" (c-a-f-compose-e-'-dot-t-x-t), and this is what the ls output looked like:
$ /bin/ls café.txt | od -tx1 -Ad -c
0000000 63 61 66 c3 a9 2e 74 78 74 0a
c a f 303 251 . t x t \n
0000010and xfs_db confirms that's what's in the file name:
# xfs_db /tmp/a.img xfs_db> sb 0 xfs_db> addr rootino xfs_db> p u3.sfdir3 u3.sfdir3.hdr.count = 1 u3.sfdir3.hdr.i8count = 0 u3.sfdir3.hdr.parent.i4 = 128 u3.sfdir3.list[0].namelen = 9 u3.sfdir3.list[0].offset = 0x60 u3.sfdir3.list[0].name = "caf\303\251.txt" u3.sfdir3.list[0].inumber.i4 = 131 u3.sfdir3.list[0].filetype = 1
Is there a particular reason you picked NFKD? Ohhh, right, because this series is a derivative of the ~2014 XFS case folding patchset. Hmm, so looking at the ext4 changes, I guess what you do is add a custom ->d_hash function so that the dentries are hashed by hash(nfkd(fname))? Which makes it easy to have link() look for names that will conflict after normalization? Which I guess is also how casefolding happens for lookups? And I suppose that's why the superblock also has to store the version of Unicode used and the folding options? Hmm, ok, sounds reasonable to me so far.
Quoted 36 lines
backward compatible. Likewise, superblock flags also define the behavior when dealing with invalid sequences. The default behavior is to just treat invalid sequences as opaque byte sequences, falling back to the original behavior. Alternatively if the strict flag is enabled, the kernel will reject invalid sequences as soon as they are detected. All these flags, can only be set at mkfs time, using the
encoding_flagsparameter.Each encoding has its default flags, when creating the filesystem in mkfs, regarding normalization/casefold functions and how to deal with opaque sequences.
The NLS patches implement generic casefold and normalization operation (defined as
tolower()andidentity(), respectively), allowing us to use any NLS charset in the kernel. We store the NLS charset as a magic number in the superblock, and for that only a few charsets are defined. If you want to force a different charset, the encoding andencoding_flagsmount options are also provided.This patchset also includes the NLS changes that I am proposing, including the entire utf8 normalization implementation. Differently from the previous version, I merged utf8 and utf8n, allowing the user to request a specific normalization (or no normalization at all) using the flags. As usual, I did not include the source ucd files because they would bounce in the list, but a completely functional implementation can be found at:
https://gitlab.collabora.com/krisman/linux.git -b ext4-ci-directory
I am also not supporting encoding with encrypted directories, given the cost of searching encrypted directories where the diggested name is not normalized. This means that we need to decrypt each filename beforehand, so I decided to simply skip this for now. If the user tries to mount with encoding a directory that has the encryption feature, we simply bail out saying it is not supported.
Quoted 6 lines
The patchset survives without failures the smoke tests of xfstests, with the obvious exception of generic/453. This test, which verifies that multiple files having equivalent names (which would match in utf8 normalization) are not the same file, doesn't really make sense with this patchset, since it basically verifies the fs is *not* encoding aware.
Yep. This probably needs a _require_unnormalized_dnames() helper to detect when filename normalization is happening. IIRC this test also fails on HFS+, as expected.
Do you normalize attr names? I surmise not since you didn't complain about generic/454.
Quoted 5 lines
We also developed encoding and casefolding tests for xfstests, allowing us to quickly verify the implementation. I will be sumitting them upstream too, but for now they are available at:
https://gitlab.collabora.com/krisman/xfstests.git -b encoding
Er... those common/encoding helpers are going to have to deal with other filesystems. :)
Quoted 151 lines
These tests validate the usage on inline dirs, on dx directories, when dealing with dcache and more. I am happy with the coverage we have now, but if you have specific concerns I can add more tests.
A modified version of e2fsprogs is necessary to run the tests, and to enable the feature. Support for mkfs, fsck, lsattr, chattr is available. I also hacked tune2fs to prevent it from setting the encryption flag when encoding is enabled. This tune2fs change is temporary until we are able to support these two features together.
https://gitlab.collabora.com/krisman/e2fsprogs.git -b encoding-feature
Gabriel Krisman Bertazi (21):
nls: Wrap uni2char/char2uni callers
nls: Wrap charset field access
nls: Wrap charset hooks in ops structure
nls: Split default charset from NLS core
nls: Split structnls_charsetfrom structnls_table
nls: Add support for multiple versions of an encoding
nls: ImplementNLS_STRICT_MODEflag
nls: Let charsets define the behavior of tolower/toupper
nls: Add new interface for string comparisons
nls: Add optional normalization and casefold hooks
nls: ascii: Support validation and normalization operations
nls: utf8: Move nls-utf8{,-core}.c
nls: utf8: Integrate utf8 normalization code with utf8 charset
nls: utf8: Introduce test module for normalized utf8 implementation
vfs: Handle case-exact lookup ind_add_ci
ext4: Include encoding information in the superblock
ext4: Add encoding mount options
ext4: Support encoding-aware file name lookups
ext4: Implement encoding-aware dcache hooks
ext4: ImplementEXT4_CASEFOLD_FLflag
docs:ext4.rst: Document encoding and case-insensitive lookupsOlaf Weber (4):
nls: utf8n: Add unicode character database files
scripts: add trie generator for UTF-8
nls: utf8: Introduce code for UTF-8 normalization
nls: utf8n: reduce the size of utf8data[]Documentation/filesystems/ext4/ext4.rst | 38 + fs/befs/linuxvfs.c | 8 +- fs/cifs/cifs_unicode.c | 15 +- fs/cifs/cifsfs.c | 2 +- fs/cifs/connect.c | 2 +- fs/cifs/dir.c | 7 +- fs/dcache.c | 33 +- fs/ext4/dir.c | 59 + fs/ext4/ext4.h | 33 +- fs/ext4/hash.c | 38 +- fs/ext4/ialloc.c | 2 +- fs/ext4/inline.c | 2 +- fs/ext4/inode.c | 4 +- fs/ext4/ioctl.c | 18 + fs/ext4/namei.c | 99 +- fs/ext4/super.c | 170 ++ fs/fat/dir.c | 13 +- fs/fat/inode.c | 6 +- fs/fat/namei_vfat.c | 6 +- fs/hfs/super.c | 6 +- fs/hfs/trans.c | 9 +- fs/hfsplus/options.c | 2 +- fs/hfsplus/unicode.c | 6 +- fs/isofs/inode.c | 5 +- fs/isofs/joliet.c | 3 +- fs/jfs/jfs_unicode.c | 9 +- fs/jfs/super.c | 3 +- fs/nls/Kconfig | 15 + fs/nls/Makefile | 20 + fs/nls/mac-celtic.c | 34 +- fs/nls/mac-centeuro.c | 34 +- fs/nls/mac-croatian.c | 34 +- fs/nls/mac-cyrillic.c | 34 +- fs/nls/mac-gaelic.c | 34 +- fs/nls/mac-greek.c | 34 +- fs/nls/mac-iceland.c | 34 +- fs/nls/mac-inuit.c | 34 +- fs/nls/mac-roman.c | 34 +- fs/nls/mac-romanian.c | 34 +- fs/nls/mac-turkish.c | 34 +- fs/nls/nls_ascii.c | 84 +- fs/nls/nls_core.c | 163 ++ fs/nls/nls_cp1250.c | 34 +- fs/nls/nls_cp1251.c | 34 +- fs/nls/nls_cp1255.c | 36 +- fs/nls/nls_cp437.c | 34 +- fs/nls/nls_cp737.c | 34 +- fs/nls/nls_cp775.c | 34 +- fs/nls/nls_cp850.c | 34 +- fs/nls/nls_cp852.c | 34 +- fs/nls/nls_cp855.c | 34 +- fs/nls/nls_cp857.c | 34 +- fs/nls/nls_cp860.c | 34 +- fs/nls/nls_cp861.c | 34 +- fs/nls/nls_cp862.c | 34 +- fs/nls/nls_cp863.c | 34 +- fs/nls/nls_cp864.c | 34 +- fs/nls/nls_cp865.c | 34 +- fs/nls/nls_cp866.c | 34 +- fs/nls/nls_cp869.c | 34 +- fs/nls/nls_cp874.c | 36 +- fs/nls/nls_cp932.c | 36 +- fs/nls/nls_cp936.c | 36 +- fs/nls/nls_cp949.c | 36 +- fs/nls/nls_cp950.c | 36 +- fs/nls/{nls_base.c => nls_default.c} | 124 +- fs/nls/nls_euc-jp.c | 29 +- fs/nls/nls_iso8859-1.c | 34 +- fs/nls/nls_iso8859-13.c | 34 +- fs/nls/nls_iso8859-14.c | 34 +- fs/nls/nls_iso8859-15.c | 34 +- fs/nls/nls_iso8859-2.c | 34 +- fs/nls/nls_iso8859-3.c | 34 +- fs/nls/nls_iso8859-4.c | 34 +- fs/nls/nls_iso8859-5.c | 34 +- fs/nls/nls_iso8859-6.c | 34 +- fs/nls/nls_iso8859-7.c | 34 +- fs/nls/nls_iso8859-9.c | 34 +- fs/nls/nls_koi8-r.c | 34 +- fs/nls/nls_koi8-ru.c | 30 +- fs/nls/nls_koi8-u.c | 34 +- fs/nls/nls_utf8-core.c | 333 +++ fs/nls/nls_utf8-norm.c | 797 ++++++ fs/nls/nls_utf8-selftest.c | 307 ++ fs/nls/nls_utf8.c | 67 - fs/nls/ucd/README | 33 + fs/nls/utf8n.h | 117 + fs/ntfs/inode.c | 2 +- fs/ntfs/super.c | 6 +- fs/ntfs/unistr.c | 13 +- fs/udf/super.c | 3 +- fs/udf/unicode.c | 4 +- include/linux/fs.h | 2 + include/linux/nls.h | 293 +- scripts/Makefile | 1 + scripts/mkutf8data.c | 3464 +++++++++++++++++++++++ 96 files changed, 7482 insertions(+), 633 deletions(-) create mode 100644 fs/nls/nls_core.c rename fs/nls/{nls_base.c => nls_default.c} (89%) create mode 100644 fs/nls/nls_utf8-core.c create mode 100644 fs/nls/nls_utf8-norm.c create mode 100644 fs/nls/nls_utf8-selftest.c delete mode 100644 fs/nls/nls_utf8.c create mode 100644 fs/nls/ucd/README create mode 100644 fs/nls/utf8n.h create mode 100644 scripts/mkutf8data.cSignature · 1 line
2.19.0
Quoted 3 lines · Darrick J. Wong
Yep. This probably needs a
_require_unnormalized_dnames()helper to detect when filename normalization is happening. IIRC this test also fails on HFS+, as expected.
Hey Darrick,
Yes, I patched my xfstests locally, but only for ext4. I Will include this test in a more generic manner when submitting the xfstests patches.
Quoted 3 lines
Do you normalize attr names? I surmise not since you didn't complain about generic/454.
No, I'm not normalizing attr names.
Quoted 159 lines
Quoted 5 lines
We also developed encoding and casefolding tests for xfstests, allowing us to quickly verify the implementation. I will be sumitting them upstream too, but for now they are available at:
https://gitlab.collabora.com/krisman/xfstests.git -b encoding
Er... those common/encoding helpers are going to have to deal with other filesystems. :)
Quoted 151 lines
These tests validate the usage on inline dirs, on dx directories, when dealing with dcache and more. I am happy with the coverage we have now, but if you have specific concerns I can add more tests.
A modified version of e2fsprogs is necessary to run the tests, and to enable the feature. Support for mkfs, fsck, lsattr, chattr is available. I also hacked tune2fs to prevent it from setting the encryption flag when encoding is enabled. This tune2fs change is temporary until we are able to support these two features together.
https://gitlab.collabora.com/krisman/e2fsprogs.git -b encoding-feature
Gabriel Krisman Bertazi (21):
nls: Wrap uni2char/char2uni callers
nls: Wrap charset field access
nls: Wrap charset hooks in ops structure
nls: Split default charset from NLS core
nls: Split structnls_charsetfrom structnls_table
nls: Add support for multiple versions of an encoding
nls: ImplementNLS_STRICT_MODEflag
nls: Let charsets define the behavior of tolower/toupper
nls: Add new interface for string comparisons
nls: Add optional normalization and casefold hooks
nls: ascii: Support validation and normalization operations
nls: utf8: Move nls-utf8{,-core}.c
nls: utf8: Integrate utf8 normalization code with utf8 charset
nls: utf8: Introduce test module for normalized utf8 implementation
vfs: Handle case-exact lookup ind_add_ci
ext4: Include encoding information in the superblock
ext4: Add encoding mount options
ext4: Support encoding-aware file name lookups
ext4: Implement encoding-aware dcache hooks
ext4: ImplementEXT4_CASEFOLD_FLflag
docs:ext4.rst: Document encoding and case-insensitive lookupsOlaf Weber (4):
nls: utf8n: Add unicode character database files
scripts: add trie generator for UTF-8
nls: utf8: Introduce code for UTF-8 normalization
nls: utf8n: reduce the size of utf8data[]Documentation/filesystems/ext4/ext4.rst | 38 + fs/befs/linuxvfs.c | 8 +- fs/cifs/cifs_unicode.c | 15 +- fs/cifs/cifsfs.c | 2 +- fs/cifs/connect.c | 2 +- fs/cifs/dir.c | 7 +- fs/dcache.c | 33 +- fs/ext4/dir.c | 59 + fs/ext4/ext4.h | 33 +- fs/ext4/hash.c | 38 +- fs/ext4/ialloc.c | 2 +- fs/ext4/inline.c | 2 +- fs/ext4/inode.c | 4 +- fs/ext4/ioctl.c | 18 + fs/ext4/namei.c | 99 +- fs/ext4/super.c | 170 ++ fs/fat/dir.c | 13 +- fs/fat/inode.c | 6 +- fs/fat/namei_vfat.c | 6 +- fs/hfs/super.c | 6 +- fs/hfs/trans.c | 9 +- fs/hfsplus/options.c | 2 +- fs/hfsplus/unicode.c | 6 +- fs/isofs/inode.c | 5 +- fs/isofs/joliet.c | 3 +- fs/jfs/jfs_unicode.c | 9 +- fs/jfs/super.c | 3 +- fs/nls/Kconfig | 15 + fs/nls/Makefile | 20 + fs/nls/mac-celtic.c | 34 +- fs/nls/mac-centeuro.c | 34 +- fs/nls/mac-croatian.c | 34 +- fs/nls/mac-cyrillic.c | 34 +- fs/nls/mac-gaelic.c | 34 +- fs/nls/mac-greek.c | 34 +- fs/nls/mac-iceland.c | 34 +- fs/nls/mac-inuit.c | 34 +- fs/nls/mac-roman.c | 34 +- fs/nls/mac-romanian.c | 34 +- fs/nls/mac-turkish.c | 34 +- fs/nls/nls_ascii.c | 84 +- fs/nls/nls_core.c | 163 ++ fs/nls/nls_cp1250.c | 34 +- fs/nls/nls_cp1251.c | 34 +- fs/nls/nls_cp1255.c | 36 +- fs/nls/nls_cp437.c | 34 +- fs/nls/nls_cp737.c | 34 +- fs/nls/nls_cp775.c | 34 +- fs/nls/nls_cp850.c | 34 +- fs/nls/nls_cp852.c | 34 +- fs/nls/nls_cp855.c | 34 +- fs/nls/nls_cp857.c | 34 +- fs/nls/nls_cp860.c | 34 +- fs/nls/nls_cp861.c | 34 +- fs/nls/nls_cp862.c | 34 +- fs/nls/nls_cp863.c | 34 +- fs/nls/nls_cp864.c | 34 +- fs/nls/nls_cp865.c | 34 +- fs/nls/nls_cp866.c | 34 +- fs/nls/nls_cp869.c | 34 +- fs/nls/nls_cp874.c | 36 +- fs/nls/nls_cp932.c | 36 +- fs/nls/nls_cp936.c | 36 +- fs/nls/nls_cp949.c | 36 +- fs/nls/nls_cp950.c | 36 +- fs/nls/{nls_base.c => nls_default.c} | 124 +- fs/nls/nls_euc-jp.c | 29 +- fs/nls/nls_iso8859-1.c | 34 +- fs/nls/nls_iso8859-13.c | 34 +- fs/nls/nls_iso8859-14.c | 34 +- fs/nls/nls_iso8859-15.c | 34 +- fs/nls/nls_iso8859-2.c | 34 +- fs/nls/nls_iso8859-3.c | 34 +- fs/nls/nls_iso8859-4.c | 34 +- fs/nls/nls_iso8859-5.c | 34 +- fs/nls/nls_iso8859-6.c | 34 +- fs/nls/nls_iso8859-7.c | 34 +- fs/nls/nls_iso8859-9.c | 34 +- fs/nls/nls_koi8-r.c | 34 +- fs/nls/nls_koi8-ru.c | 30 +- fs/nls/nls_koi8-u.c | 34 +- fs/nls/nls_utf8-core.c | 333 +++ fs/nls/nls_utf8-norm.c | 797 ++++++ fs/nls/nls_utf8-selftest.c | 307 ++ fs/nls/nls_utf8.c | 67 - fs/nls/ucd/README | 33 + fs/nls/utf8n.h | 117 + fs/ntfs/inode.c | 2 +- fs/ntfs/super.c | 6 +- fs/ntfs/unistr.c | 13 +- fs/udf/super.c | 3 +- fs/udf/unicode.c | 4 +- include/linux/fs.h | 2 + include/linux/nls.h | 293 +- scripts/Makefile | 1 + scripts/mkutf8data.c | 3464 +++++++++++++++++++++++ 96 files changed, 7482 insertions(+), 633 deletions(-) create mode 100644 fs/nls/nls_core.c rename fs/nls/{nls_base.c => nls_default.c} (89%) create mode 100644 fs/nls/nls_utf8-core.c create mode 100644 fs/nls/nls_utf8-norm.c create mode 100644 fs/nls/nls_utf8-selftest.c delete mode 100644 fs/nls/nls_utf8.c create mode 100644 fs/nls/ucd/README create mode 100644 fs/nls/utf8n.h create mode 100644 scripts/mkutf8data.cSignature · 1 line
2.19.0
Signature · 1 line
Gabriel Krisman Bertazi
Theodore Y. Ts'o
maintainerQuoted 10 lines · Darrick J. Wong
Hmmm, I'm curious, why pick NFKD specifically? AFAICT Linux userspace environments (I only tried with GNOME and KDE) use NF[K]C....
Is there a particular reason you picked NFKD? Ohhh, right, because this series is a derivative of the ~2014 XFS case folding patchset. Hmm, so looking at the ext4 changes, I guess what you do is add a custom ->d_hash function so that the dentries are hashed by hash(nfkd(fname))? Which makes it easy to have
link()look for names that will conflict after normalization?
This would be true for NFKC or NFC as well though, right? So the tradeoff of NF[K]C vs NF[K]D is that NFC is more efficient from an encoding perspective. For e with a grave accent, NFC would encode it as C3 A9, while NFD would encode it as 65 CC 81. So from an encoding perspective there would be a benefit to use 'C' versus 'D'. But MacOS X by default canonicalizes to 'D', not 'C'. I assume that's the rationale for using NFKD versus NFKC?
As far as the 'K' versus "non-K" distinction, I imagine the main issue is that a user could cut and paste something like "She\uFB03eld" which it makes sense to canonicalize this to "Sheffield". This is *not* a canoncalization which MacOS X does (it uses NFD, not NFKD) but from a compatibility perspective, it's not a problem since:
NFD: Sheffield -> Sheffield
She\uFB03eld -> She\uFB03eld
NFKD: Sheffield -> Sheffield
She\uFB03eld -> Sheffield
Given it's really painful to type the string She\uFB03eld into a terminal, it seems to make sense that even if the user tries to create a file with that string, that the actual file name that should get created should be "Sheffield".
And hence, that's the argument for why the best on-disk encoding for Linux file systems should be NFKD.
Does that seem right to everyone?
- Ted