Skip to contentSkip to conversation

Ext4 Encoding and Case-insensitive support

PATCH Superseded Gabriel Krisman Bertazi 5 participants 36 messages Dec 6, 2018 – Dec 10, 2018 2 messages not in the archive

Subsystems (12)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover Ext4 Encoding and Case-insensitive support

Hi,

[Resending to include fsdevel, as requested by Dave Chinner]

Following the e2fsprogs changes, these are the corresponding kernel-side modifications to support the fname_encoding feature.

The patches are split in two parts. The fist 14 patches are refactoring and improvements to the NLS code, including the utf8 normalization support. The final patches implement the fname_encoding feature in ext4.

To test this feature, you need to use the tip of e2fsprogs branch, which already include support for enabling this feature.

As usual, the ucd files are not included in this email because they are too large, and would actually cause the email message to bounce.

There are two test files for this in a private xfstests branch, that I plan to submit upstream once we get this series merged:

https://gitlab.collabora.com/krisman/xfstests.git -b encoding_v4

I also tested this with the xfstests smoke tests using two scenarios:
(1) a non-encoding TEST_DEV; (2) a utf8 enabled TEST_DEV. On both cases, no unrelated regressions where observed. With my branch of xfstests above, that fixes some related tests, I didn't observe any regressions.

Gabriel Krisman Bertazi (19):
nls: Wrap uni2char/char2uni callers
nls: Wrap charset field access
nls: Wrap charset hooks in ops structure
nls: Split default charset from NLS core
nls: Split struct nls_charset from struct nls_table
nls: Add support for multiple versions of an encoding
nls: Implement NLS_STRICT_MODE flag
nls: Let charsets define the behavior of tolower/toupper
nls: Add new interface for string comparisons
nls: Add optional normalization and casefold hooks
nls: ascii: Support validation and normalization operations
nls: utf8: Move nls-utf8{,-core}.c
nls: utf8: Integrate utf8 normalization code with utf8 charset
nls: utf8: Introduce test module for normalized utf8 implementation
ext4: Reserve superblock fields for encoding information
ext4: Include encoding information in the superblock
ext4: Support encoding-aware file name lookups
ext4: Implement EXT4_CASEFOLD_FL flag
docs: ext4.rst: Document encoding and case-insensitive

Olaf Weber (4):
nls: utf8: Add unicode character database files
scripts: add trie generator for UTF-8
nls: utf8: Introduce code for UTF-8 normalization
nls: utf8n: reduce the size of utf8data[]

 Documentation/admin-guide/ext4.rst   |   29 +
 fs/befs/linuxvfs.c                   |    8 +-
 fs/cifs/cifs_unicode.c               |   15 +-
 fs/cifs/cifsfs.c                     |    2 +-
 fs/cifs/connect.c                    |    2 +-
 fs/cifs/dir.c                        |    7 +-
 fs/ext4/dir.c                        |   59 +
 fs/ext4/ext4.h                       |   33 +-
 fs/ext4/hash.c                       |   38 +-
 fs/ext4/ialloc.c                     |    2 +-
 fs/ext4/inline.c                     |    2 +-
 fs/ext4/inode.c                      |    4 +-
 fs/ext4/ioctl.c                      |   18 +
 fs/ext4/namei.c                      |   85 +-
 fs/ext4/super.c                      |   83 +
 fs/fat/dir.c                         |   13 +-
 fs/fat/inode.c                       |    6 +-
 fs/fat/namei_vfat.c                  |    6 +-
 fs/hfs/super.c                       |    6 +-
 fs/hfs/trans.c                       |    9 +-
 fs/hfsplus/options.c                 |    2 +-
 fs/hfsplus/unicode.c                 |    6 +-
 fs/isofs/inode.c                     |    5 +-
 fs/isofs/joliet.c                    |    3 +-
 fs/jfs/jfs_unicode.c                 |    9 +-
 fs/jfs/super.c                       |    3 +-
 fs/nls/Kconfig                       |   15 +
 fs/nls/Makefile                      |   20 +
 fs/nls/mac-celtic.c                  |   34 +-
 fs/nls/mac-centeuro.c                |   34 +-
 fs/nls/mac-croatian.c                |   34 +-
 fs/nls/mac-cyrillic.c                |   34 +-
 fs/nls/mac-gaelic.c                  |   34 +-
 fs/nls/mac-greek.c                   |   34 +-
 fs/nls/mac-iceland.c                 |   34 +-
 fs/nls/mac-inuit.c                   |   34 +-
 fs/nls/mac-roman.c                   |   34 +-
 fs/nls/mac-romanian.c                |   34 +-
 fs/nls/mac-turkish.c                 |   34 +-
 fs/nls/nls_ascii.c                   |   84 +-
 fs/nls/nls_core.c                    |  163 ++
 fs/nls/nls_cp1250.c                  |   34 +-
 fs/nls/nls_cp1251.c                  |   34 +-
 fs/nls/nls_cp1255.c                  |   36 +-
 fs/nls/nls_cp437.c                   |   34 +-
 fs/nls/nls_cp737.c                   |   34 +-
 fs/nls/nls_cp775.c                   |   34 +-
 fs/nls/nls_cp850.c                   |   34 +-
 fs/nls/nls_cp852.c                   |   34 +-
 fs/nls/nls_cp855.c                   |   34 +-
 fs/nls/nls_cp857.c                   |   34 +-
 fs/nls/nls_cp860.c                   |   34 +-
 fs/nls/nls_cp861.c                   |   34 +-
 fs/nls/nls_cp862.c                   |   34 +-
 fs/nls/nls_cp863.c                   |   34 +-
 fs/nls/nls_cp864.c                   |   34 +-
 fs/nls/nls_cp865.c                   |   34 +-
 fs/nls/nls_cp866.c                   |   34 +-
 fs/nls/nls_cp869.c                   |   34 +-
 fs/nls/nls_cp874.c                   |   36 +-
 fs/nls/nls_cp932.c                   |   36 +-
 fs/nls/nls_cp936.c                   |   36 +-
 fs/nls/nls_cp949.c                   |   36 +-
 fs/nls/nls_cp950.c                   |   36 +-
 fs/nls/{nls_base.c => nls_default.c} |  124 +-
 fs/nls/nls_euc-jp.c                  |   29 +-
 fs/nls/nls_iso8859-1.c               |   34 +-
 fs/nls/nls_iso8859-13.c              |   34 +-
 fs/nls/nls_iso8859-14.c              |   34 +-
 fs/nls/nls_iso8859-15.c              |   34 +-
 fs/nls/nls_iso8859-2.c               |   34 +-
 fs/nls/nls_iso8859-3.c               |   34 +-
 fs/nls/nls_iso8859-4.c               |   34 +-
 fs/nls/nls_iso8859-5.c               |   34 +-
 fs/nls/nls_iso8859-6.c               |   34 +-
 fs/nls/nls_iso8859-7.c               |   34 +-
 fs/nls/nls_iso8859-9.c               |   34 +-
 fs/nls/nls_koi8-r.c                  |   34 +-
 fs/nls/nls_koi8-ru.c                 |   30 +-
 fs/nls/nls_koi8-u.c                  |   34 +-
 fs/nls/nls_utf8-core.c               |  328 +++
 fs/nls/nls_utf8-norm.c               |  797 ++++++
 fs/nls/nls_utf8-selftest.c           |  316 +++
 fs/nls/nls_utf8.c                    |   67 -
 fs/nls/ucd/README                    |   34 +
 fs/nls/utf8n.h                       |  117 +
 fs/ntfs/inode.c                      |    2 +-
 fs/ntfs/super.c                      |    6 +-
 fs/ntfs/unistr.c                     |   13 +-
 fs/udf/super.c                       |    3 +-
 fs/udf/unicode.c                     |    4 +-
 include/linux/fs.h                   |    2 +
 include/linux/nls.h                  |  293 ++-
 scripts/Makefile                     |    1 +
 scripts/mkutf8data.c                 | 3392 ++++++++++++++++++++++++++
 95 files changed, 7287 insertions(+), 618 deletions(-)
 create mode 100644 fs/nls/nls_core.c
 rename fs/nls/{nls_base.c => nls_default.c} (89%)
 create mode 100644 fs/nls/nls_utf8-core.c
 create mode 100644 fs/nls/nls_utf8-norm.c
 create mode 100644 fs/nls/nls_utf8-selftest.c
 delete mode 100644 fs/nls/nls_utf8.c
 create mode 100644 fs/nls/ucd/README
 create mode 100644 fs/nls/utf8n.h
 create mode 100644 scripts/mkutf8data.c
Signature · 1 line
2.20.0.rc2
#1 lore raw

PATCH v4 1/23 nls: Wrap uni2char/char2uni callers

From: Gabriel Krisman Bertazi <[email protected]>

Just a cosmetic change at this point, this patch will simplify the following Coccinelle patches which will move the hooks into a dedicated structure, with the goal of splitting the nls_table structure to support versioning.

This was generated with the following coccinele script:

<smpl>

@@
expression A, B, C, D;
@@
(
- A->uni2char(B, C, D)
+ nls_uni2char(A, B, C, D)
|
- A->char2uni(B, C, D)
+ nls_char2uni(A, B, C, D)
)

</smpl>

---
 fs/befs/linuxvfs.c     |  4 ++--
 fs/cifs/cifs_unicode.c |  9 +++++----
 fs/cifs/dir.c          |  7 ++++---
 fs/fat/dir.c           |  8 ++++----
 fs/fat/namei_vfat.c    |  6 +++---
 fs/hfs/trans.c         |  9 +++++----
 fs/hfsplus/unicode.c   |  6 +++---
 fs/isofs/joliet.c      |  3 ++-
 fs/jfs/jfs_unicode.c   |  7 +++----
 fs/nls/nls_euc-jp.c    |  4 ++--
 fs/nls/nls_koi8-ru.c   |  6 +++---
 fs/ntfs/unistr.c       |  8 ++++----
 include/linux/nls.h    | 13 +++++++++++++
 13 files changed, 53 insertions(+), 37 deletions(-)
fs/befs/linuxvfs.c +2 -2
index 4700b4534439..0ba368fbfad4 100644
@@ -542,7 +542,7 @@ befs_utf2nls(struct super_block *sb, const char *in,
 		/* convert from Unicode to nls */
 		if (uni > MAX_WCHAR_T)
 			goto conv_err;
-		unilen = nls->uni2char(uni, &result[o], in_len - o);
+		unilen = nls_uni2char(nls, uni, &result[o], in_len - o);
 		if (unilen < 0)
 			goto conv_err;
 	}
@@ -616,7 +616,7 @@ befs_nls2utf(struct super_block *sb, const char *in,
 	for (i = o = 0; i < in_len; i += unilen, o += utflen) {
 
 		/* convert from nls to unicode */
-		unilen = nls->char2uni(&in[i], in_len - i, &uni);
+		unilen = nls_char2uni(nls, &in[i], in_len - i, &uni);
 		if (unilen < 0)
 			goto conv_err;
 
fs/cifs/cifs_unicode.c +5 -4
index a2b2355e7f01..ffad8b4f90d1 100644
@@ -145,7 +145,7 @@ cifs_mapchar(char *target, const __u16 *from, const struct nls_table *cp,
 		return len;
 
 	/* if character not one of seven in special remap set */
-	len = cp->uni2char(src_char, target, NLS_MAX_CHARSET_SIZE);
+	len = nls_uni2char(cp, src_char, target, NLS_MAX_CHARSET_SIZE);
 	if (len <= 0)
 		goto surrogate_pair;
 
@@ -289,7 +289,7 @@ cifs_strtoUTF16(__le16 *to, const char *from, int len,
 	}
 
 	for (i = 0; len && *from; i++, from += charlen, len -= charlen) {
-		charlen = codepage->char2uni(from, len, &wchar_to);
+		charlen = nls_char2uni(codepage, from, len, &wchar_to);
 		if (charlen < 1) {
 			cifs_dbg(VFS, "strtoUTF16: char2uni of 0x%x returned %d\n",
 				 *from, charlen);
@@ -515,7 +515,8 @@ cifsConvertToUTF16(__le16 *target, const char *source, int srclen,
 		 * as they use backslash as separator.
 		 */
 		if (dst_char == 0) {
-			charlen = cp->char2uni(source + i, srclen - i, &tmp);
+			charlen = nls_char2uni(cp, source + i, srclen - i,
+					       &tmp);
 			dst_char = cpu_to_le16(tmp);
 
 			/*
@@ -605,7 +606,7 @@ cifs_local_to_utf16_bytes(const char *from, int len,
 	wchar_t wchar_to;
 
 	for (i = 0; len && *from; i++, from += charlen, len -= charlen) {
-		charlen = codepage->char2uni(from, len, &wchar_to);
+		charlen = nls_char2uni(codepage, from, len, &wchar_to);
 		/* Failed conversion defaults to a question mark */
 		if (charlen < 1)
 			charlen = 1;
fs/cifs/dir.c +4 -3
index 3713d22b95a7..f8bb9285f630 100644
@@ -910,7 +910,7 @@ static int cifs_ci_hash(const struct dentry *dentry, struct qstr *q)
 
 	hash = init_name_hash(dentry);
 	for (i = 0; i < q->len; i += charlen) {
-		charlen = codepage->char2uni(&q->name[i], q->len - i, &c);
+		charlen = nls_char2uni(codepage, &q->name[i], q->len - i, &c);
 		/* error out if we can't convert the character */
 		if (unlikely(charlen < 0))
 			return charlen;
@@ -939,8 +939,9 @@ static int cifs_ci_compare(const struct dentry *dentry,
 
 	for (i = 0; i < len; i += l1) {
 		/* Convert characters in both strings to UTF-16. */
-		l1 = codepage->char2uni(&str[i], len - i, &c1);
-		l2 = codepage->char2uni(&name->name[i], name->len - i, &c2);
+		l1 = nls_char2uni(codepage, &str[i], len - i, &c1);
+		l2 = nls_char2uni(codepage, &name->name[i], name->len - i,
+				  &c2);
 
 		/*
 		 * If we can't convert either character, just declare it to
fs/fat/dir.c +4 -4
index c8366cb8eccd..d5f856651a08 100644
@@ -153,7 +153,7 @@ static int uni16_to_x8(struct super_block *sb, unsigned char *ascii,
 
 	while (*ip && ((len - NLS_MAX_CHARSET_SIZE) > 0)) {
 		ec = *ip++;
-		charlen = nls->uni2char(ec, op, NLS_MAX_CHARSET_SIZE);
+		charlen = nls_uni2char(nls, ec, op, NLS_MAX_CHARSET_SIZE);
 		if (charlen > 0) {
 			op += charlen;
 			len -= charlen;
@@ -195,7 +195,7 @@ fat_short2uni(struct nls_table *t, unsigned char *c, int clen, wchar_t *uni)
 {
 	int charlen;
 
-	charlen = t->char2uni(c, clen, uni);
+	charlen = nls_char2uni(t, c, clen, uni);
 	if (charlen < 0) {
 		*uni = 0x003f;	/* a question mark */
 		charlen = 1;
@@ -210,7 +210,7 @@ fat_short2lower_uni(struct nls_table *t, unsigned char *c,
 	int charlen;
 	wchar_t wc;
 
-	charlen = t->char2uni(c, clen, &wc);
+	charlen = nls_char2uni(t, c, clen, &wc);
 	if (charlen < 0) {
 		*uni = 0x003f;	/* a question mark */
 		charlen = 1;
@@ -220,7 +220,7 @@ fat_short2lower_uni(struct nls_table *t, unsigned char *c,
 		if (!nc)
 			nc = *c;
 
-		charlen = t->char2uni(&nc, 1, uni);
+		charlen = nls_char2uni(t, &nc, 1, uni);
 		if (charlen < 0) {
 			*uni = 0x003f;	/* a question mark */
 			charlen = 1;
fs/fat/namei_vfat.c +3 -3
index 996c8c25e9c6..ab6b450521ff 100644
@@ -289,7 +289,7 @@ static inline int to_shortname_char(struct nls_table *nls,
 		return 1;
 	}
 
-	len = nls->uni2char(*src, buf, buf_size);
+	len = nls_uni2char(nls, *src, buf, buf_size);
 	if (len <= 0) {
 		info->valid = 0;
 		buf[0] = '_';
@@ -544,8 +544,8 @@ xlate_to_uni(const unsigned char *name, int len, unsigned char *outname,
 				ip += 5;
 				i += 5;
 			} else {
-				charlen = nls->char2uni(ip, len - i,
-							(wchar_t *)op);
+				charlen = nls_char2uni(nls, ip, len - i,
+						       (wchar_t *)op);
 				if (charlen < 0)
 					return -EINVAL;
 				ip += charlen;
fs/hfs/trans.c +5 -4
index 39f5e343bf4d..2fae312edb31 100644
@@ -49,7 +49,8 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in)
 
 		while (srclen > 0) {
 			if (nls_disk) {
-				size = nls_disk->char2uni(src, srclen, &ch);
+				size = nls_char2uni(nls_disk, src, srclen,
+						    &ch);
 				if (size <= 0) {
 					ch = '?';
 					size = 1;
@@ -62,7 +63,7 @@ int hfs_mac2asc(struct super_block *sb, char *out, const struct hfs_name *in)
 			}
 			if (ch == '/')
 				ch = ':';
-			size = nls_io->uni2char(ch, dst, dstlen);
+			size = nls_uni2char(nls_io, ch, dst, dstlen);
 			if (size < 0) {
 				if (size == -ENAMETOOLONG)
 					goto out;
@@ -110,7 +111,7 @@ void hfs_asc2mac(struct super_block *sb, struct hfs_name *out, const struct qstr
 		wchar_t ch;
 
 		while (srclen > 0) {
-			size = nls_io->char2uni(src, srclen, &ch);
+			size = nls_char2uni(nls_io, src, srclen, &ch);
 			if (size < 0) {
 				ch = '?';
 				size = 1;
@@ -120,7 +121,7 @@ void hfs_asc2mac(struct super_block *sb, struct hfs_name *out, const struct qstr
 			if (ch == ':')
 				ch = '/';
 			if (nls_disk) {
-				size = nls_disk->uni2char(ch, dst, dstlen);
+				size = nls_uni2char(nls_disk, ch, dst, dstlen);
 				if (size < 0) {
 					if (size == -ENAMETOOLONG)
 						goto out;
fs/hfsplus/unicode.c +3 -3
index c8d1b2be7854..057dc7e57cb1 100644
@@ -190,7 +190,7 @@ int hfsplus_uni2asc(struct super_block *sb,
 				c0 = ':';
 				break;
 			}
-			res = nls->uni2char(c0, op, len);
+			res = nls_uni2char(nls, c0, op, len);
 			if (res < 0) {
 				if (res == -ENAMETOOLONG)
 					goto out;
@@ -233,7 +233,7 @@ int hfsplus_uni2asc(struct super_block *sb,
 			cc = c0;
 		}
 done:
-		res = nls->uni2char(cc, op, len);
+		res = nls_uni2char(nls, cc, op, len);
 		if (res < 0) {
 			if (res == -ENAMETOOLONG)
 				goto out;
@@ -256,7 +256,7 @@ int hfsplus_uni2asc(struct super_block *sb,
 static inline int asc2unichar(struct super_block *sb, const char *astr, int len,
 			      wchar_t *uc)
 {
-	int size = HFSPLUS_SB(sb)->nls->char2uni(astr, len, uc);
+	int size = nls_char2uni(HFSPLUS_SB(sb)->nls, astr, len, uc);
 	if (size <= 0) {
 		*uc = '?';
 		size = 1;
fs/isofs/joliet.c +2 -1
index be8b6a9d0b92..56fac73b27a5 100644
@@ -25,7 +25,8 @@ uni16_to_x8(unsigned char *ascii, __be16 *uni, int len, struct nls_table *nls)
 
 	while ((ch = get_unaligned(ip)) && len) {
 		int llen;
-		llen = nls->uni2char(be16_to_cpu(ch), op, NLS_MAX_CHARSET_SIZE);
+		llen = nls_uni2char(nls, be16_to_cpu(ch), op,
+				    NLS_MAX_CHARSET_SIZE);
 		if (llen > 0)
 			op += llen;
 		else
fs/jfs/jfs_unicode.c +3 -4
index 0148e2e4d97a..4ca88ef661e9 100644
@@ -41,9 +41,8 @@ int jfs_strfromUCS_le(char *to, const __le16 * from,
 		for (i = 0; (i < len) && from[i]; i++) {
 			int charlen;
 			charlen =
-			    codepage->uni2char(le16_to_cpu(from[i]),
-					       &to[outlen],
-					       NLS_MAX_CHARSET_SIZE);
+			    nls_uni2char(codepage, le16_to_cpu(from[i]),
+					 &to[outlen], NLS_MAX_CHARSET_SIZE);
 			if (charlen > 0)
 				outlen += charlen;
 			else
@@ -88,7 +87,7 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len,
 	if (codepage) {
 		for (i = 0; len && *from; i++, from += charlen, len -= charlen)
 		{
-			charlen = codepage->char2uni(from, len, &to[i]);
+			charlen = nls_char2uni(codepage, from, len, &to[i]);
 			if (charlen < 1) {
 				jfs_err("jfs_strtoUCS: char2uni returned %d.",
 					charlen);
fs/nls/nls_euc-jp.c +2 -2
index 162b3f160353..eec257545f04 100644
@@ -413,7 +413,7 @@ static int uni2char(const wchar_t uni,
 
 	if (!p_nls)
 		return -EINVAL;
-	if ((n = p_nls->uni2char(uni, out, boundlen)) < 0)
+	if ((n = nls_uni2char(p_nls, uni, out, boundlen)) < 0)
 		return n;
 
 	/* translate SJIS into EUC-JP */
@@ -543,7 +543,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 		sjis_temp[1] = 0x00;
 	}
 
-	if ( (n = p_nls->char2uni(sjis_temp, sizeof(sjis_temp), uni)) < 0)
+	if ( (n = nls_char2uni(p_nls, sjis_temp, sizeof(sjis_temp), uni)) < 0)
 		return n;
 
 	return euc_offset;
fs/nls/nls_koi8-ru.c +3 -3
index a80a741a8676..32781252110d 100644
@@ -28,12 +28,12 @@ static int uni2char(const wchar_t uni,
 		else if (uni == 0x255d || uni == 0x256c)
 			return 0;
 		else
-			return p_nls->uni2char(uni, out, boundlen);
+			return nls_uni2char(p_nls, uni, out, boundlen);
 		return 1;
 	}
 	else
 		/* fast path */
-		return p_nls->uni2char(uni, out, boundlen);
+		return nls_uni2char(p_nls, uni, out, boundlen);
 }
 
 static int char2uni(const unsigned char *rawstring, int boundlen,
@@ -47,7 +47,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 		return 1;
 	}
 
-	n = p_nls->char2uni(rawstring, boundlen, uni);
+	n = nls_char2uni(p_nls, rawstring, boundlen, uni);
 	return n;
 }
 
fs/ntfs/unistr.c +4 -4
index 005ca4b0f132..e0a5f33441df 100644
@@ -269,8 +269,8 @@ int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,
 		ucs = kmem_cache_alloc(ntfs_name_cache, GFP_NOFS);
 		if (likely(ucs)) {
 			for (i = o = 0; i < ins_len; i += wc_len) {
-				wc_len = nls->char2uni(ins + i, ins_len - i,
-						&wc);
+				wc_len = nls_char2uni(nls, ins + i,
+						      ins_len - i, &wc);
 				if (likely(wc_len >= 0 &&
 						o < NTFS_MAX_NAME_LEN)) {
 					if (likely(wc)) {
@@ -355,8 +355,8 @@ int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins,
 				goto mem_err_out;
 		}
 		for (i = o = 0; i < ins_len; i++) {
-retry:			wc = nls->uni2char(le16_to_cpu(ins[i]), ns + o,
-					ns_len - o);
+retry:			wc = nls_uni2char(nls, le16_to_cpu(ins[i]),
+						ns + o, ns_len - o);
 			if (wc > 0) {
 				o += wc;
 				continue;
include/linux/nls.h +13 -0
index 499e486b3722..5073ecd57279 100644
@@ -59,6 +59,19 @@ extern int utf8s_to_utf16s(const u8 *s, int len,
 extern int utf16s_to_utf8s(const wchar_t *pwcs, int len,
 		enum utf16_endian endian, u8 *s, int maxlen);
 
+static inline int nls_uni2char(const struct nls_table *table, wchar_t uni,
+			       unsigned char *out, int boundlen)
+{
+	return table->uni2char(uni, out, boundlen);
+}
+
+static inline int nls_char2uni(const struct nls_table *table,
+			       const unsigned char *rawstring,
+			       int boundlen, wchar_t *uni)
+{
+	return table->char2uni(rawstring, boundlen, uni);
+}
+
 static inline unsigned char nls_tolower(struct nls_table *t, unsigned char c)
 {
 	unsigned char nc = t->charset2lower[c];
Signature · 1 line
2.20.0.rc2
#2 lore raw

PATCH v4 2/23 nls: Wrap charset field access

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

The goal is to simplify the following patches that split nls_table. No behavior changes intended.

<smpl>

@@
struct nls_table *c;
@@

- c->charset
+ nls_charset_name(c)

</smpl>

---
 fs/befs/linuxvfs.c     | 4 ++--
 fs/cifs/cifs_unicode.c | 6 +++---
 fs/cifs/cifsfs.c       | 2 +-
 fs/cifs/connect.c      | 2 +-
 fs/fat/inode.c         | 6 ++++--
 fs/hfs/super.c         | 6 ++++--
 fs/hfsplus/options.c   | 2 +-
 fs/isofs/inode.c       | 5 +++--
 fs/jfs/jfs_unicode.c   | 2 +-
 fs/jfs/super.c         | 3 ++-
 fs/nls/nls_base.c      | 2 +-
 fs/ntfs/inode.c        | 2 +-
 fs/ntfs/super.c        | 6 +++---
 fs/ntfs/unistr.c       | 5 +++--
 fs/udf/super.c         | 3 ++-
 include/linux/nls.h    | 5 +++++
 16 files changed, 37 insertions(+), 24 deletions(-)
fs/befs/linuxvfs.c +2 -2
index 0ba368fbfad4..8b7af0a9011a 100644
@@ -555,7 +555,7 @@ befs_utf2nls(struct super_block *sb, const char *in,
 
 conv_err:
 	befs_error(sb, "Name using character set %s contains a character that "
-		   "cannot be converted to unicode.", nls->charset);
+		   "cannot be converted to unicode.", nls_charset_name(nls));
 	befs_debug(sb, "<--- %s", __func__);
 	kfree(result);
 	return -EILSEQ;
@@ -635,7 +635,7 @@ befs_nls2utf(struct super_block *sb, const char *in,
 
 conv_err:
 	befs_error(sb, "Name using character set %s contains a character that "
-		   "cannot be converted to unicode.", nls->charset);
+		   "cannot be converted to unicode.", nls_charset_name(nls));
 	befs_debug(sb, "<--- %s", __func__);
 	kfree(result);
 	return -EILSEQ;
fs/cifs/cifs_unicode.c +3 -3
index ffad8b4f90d1..2a9396d24f60 100644
@@ -153,7 +153,7 @@ cifs_mapchar(char *target, const __u16 *from, const struct nls_table *cp,
 
 surrogate_pair:
 	/* convert SURROGATE_PAIR and IVS */
-	if (strcmp(cp->charset, "utf8"))
+	if (strcmp(nls_charset_name(cp), "utf8"))
 		goto unknown;
 	len = utf16s_to_utf8s(from, 3, UTF16_LITTLE_ENDIAN, target, 6);
 	if (len <= 0)
@@ -268,7 +268,7 @@ cifs_strtoUTF16(__le16 *to, const char *from, int len,
 	wchar_t wchar_to; /* needed to quiet sparse */
 
 	/* special case for utf8 to handle no plane0 chars */
-	if (!strcmp(codepage->charset, "utf8")) {
+	if (!strcmp(nls_charset_name(codepage), "utf8")) {
 		/*
 		 * convert utf8 -> utf16, we assume we have enough space
 		 * as caller should have assumed conversion does not overflow
@@ -527,7 +527,7 @@ cifsConvertToUTF16(__le16 *target, const char *source, int srclen,
 				goto ctoUTF16;
 
 			/* convert SURROGATE_PAIR */
-			if (strcmp(cp->charset, "utf8") || !wchar_to)
+			if (strcmp(nls_charset_name(cp), "utf8") || !wchar_to)
 				goto unknown;
 			if (*(source + i) & 0x80) {
 				charlen = utf8_to_utf32(source + i, 6, &u);
fs/cifs/cifsfs.c +1 -1
index 865706edb307..b0531986cfd7 100644
@@ -414,7 +414,7 @@ cifs_show_nls(struct seq_file *s, struct nls_table *cur)
 	/* Display iocharset= option if it's not default charset */
 	def = load_nls_default();
 	if (def != cur)
-		seq_printf(s, ",iocharset=%s", cur->charset);
+		seq_printf(s, ",iocharset=%s", nls_charset_name(cur));
 	unload_nls(def);
 }
 
fs/cifs/connect.c +1 -1
index 6f24f129a751..3011276a06f0 100644
@@ -3191,7 +3191,7 @@ compare_mount_options(struct super_block *sb, struct cifs_mnt_data *mnt_data)
 	    old->mnt_dir_mode != new->mnt_dir_mode)
 		return 0;
 
-	if (strcmp(old->local_nls->charset, new->local_nls->charset))
+	if (strcmp(nls_charset_name(old->local_nls), nls_charset_name(new->local_nls)))
 		return 0;
 
 	if (old->actimeo != new->actimeo)
fs/fat/inode.c +4 -2
index c0b5b5c3373b..2563dc306e7f 100644
@@ -948,10 +948,12 @@ static int fat_show_options(struct seq_file *m, struct dentry *root)
 		seq_printf(m, ",allow_utime=%04o", opts->allow_utime);
 	if (sbi->nls_disk)
 		/* strip "cp" prefix from displayed option */
-		seq_printf(m, ",codepage=%s", &sbi->nls_disk->charset[2]);
+		seq_printf(m, ",codepage=%s",
+			   &nls_charset_name(sbi->nls_disk)[2]);
 	if (isvfat) {
 		if (sbi->nls_io)
-			seq_printf(m, ",iocharset=%s", sbi->nls_io->charset);
+			seq_printf(m, ",iocharset=%s",
+				   nls_charset_name(sbi->nls_io));
 
 		switch (opts->shortname) {
 		case VFAT_SFN_DISPLAY_WIN95 | VFAT_SFN_CREATE_WIN95:
fs/hfs/super.c +4 -2
index 173876782f73..b16ca01180a5 100644
@@ -151,9 +151,11 @@ static int hfs_show_options(struct seq_file *seq, struct dentry *root)
 	if (sbi->session >= 0)
 		seq_printf(seq, ",session=%u", sbi->session);
 	if (sbi->nls_disk)
-		seq_printf(seq, ",codepage=%s", sbi->nls_disk->charset);
+		seq_printf(seq, ",codepage=%s",
+			   nls_charset_name(sbi->nls_disk));
 	if (sbi->nls_io)
-		seq_printf(seq, ",iocharset=%s", sbi->nls_io->charset);
+		seq_printf(seq, ",iocharset=%s",
+			   nls_charset_name(sbi->nls_io));
 	if (sbi->s_quiet)
 		seq_printf(seq, ",quiet");
 	return 0;
fs/hfsplus/options.c +1 -1
index 047e05c57560..2d6644465566 100644
@@ -230,7 +230,7 @@ int hfsplus_show_options(struct seq_file *seq, struct dentry *root)
 	if (sbi->session >= 0)
 		seq_printf(seq, ",session=%u", sbi->session);
 	if (sbi->nls)
-		seq_printf(seq, ",nls=%s", sbi->nls->charset);
+		seq_printf(seq, ",nls=%s", nls_charset_name(sbi->nls));
 	if (test_bit(HFSPLUS_SB_NODECOMPOSE, &sbi->flags))
 		seq_puts(seq, ",nodecompose");
 	if (test_bit(HFSPLUS_SB_NOBARRIER, &sbi->flags))
fs/isofs/inode.c +3 -2
index 488a9e7f8f66..b23a3955b8c6 100644
@@ -520,8 +520,9 @@ static int isofs_show_options(struct seq_file *m, struct dentry *root)
 
 #ifdef CONFIG_JOLIET
 	if (sbi->s_nls_iocharset &&
-	    strcmp(sbi->s_nls_iocharset->charset, CONFIG_NLS_DEFAULT) != 0)
-		seq_printf(m, ",iocharset=%s", sbi->s_nls_iocharset->charset);
+	    strcmp(nls_charset_name(sbi->s_nls_iocharset), CONFIG_NLS_DEFAULT) != 0)
+		seq_printf(m, ",iocharset=%s",
+			   nls_charset_name(sbi->s_nls_iocharset));
 #endif
 	return 0;
 }
fs/jfs/jfs_unicode.c +1 -1
index 4ca88ef661e9..1e89b3b8caa7 100644
@@ -92,7 +92,7 @@ static int jfs_strtoUCS(wchar_t * to, const unsigned char *from, int len,
 				jfs_err("jfs_strtoUCS: char2uni returned %d.",
 					charlen);
 				jfs_err("charset = %s, char = 0x%x",
-					codepage->charset, *from);
+					nls_charset_name(codepage), *from);
 				return charlen;
 			}
 		}
fs/jfs/super.c +2 -1
index 65d8fc87ab11..a04ff4bc5afd 100644
@@ -736,7 +736,8 @@ static int jfs_show_options(struct seq_file *seq, struct dentry *root)
 	if (sbi->flag & JFS_DISCARD)
 		seq_printf(seq, ",discard=%u", sbi->minblks_trim);
 	if (sbi->nls_tab)
-		seq_printf(seq, ",iocharset=%s", sbi->nls_tab->charset);
+		seq_printf(seq, ",iocharset=%s",
+			   nls_charset_name(sbi->nls_tab));
 	if (sbi->flag & JFS_ERR_CONTINUE)
 		seq_printf(seq, ",errors=continue");
 	if (sbi->flag & JFS_ERR_PANIC)
fs/nls/nls_base.c +1 -1
index 52ccd34b1e79..e5d083b6e2b2 100644
@@ -277,7 +277,7 @@ static struct nls_table *find_nls(char *charset)
 	struct nls_table *nls;
 	spin_lock(&nls_lock);
 	for (nls = tables; nls; nls = nls->next) {
-		if (!strcmp(nls->charset, charset))
+		if (!strcmp(nls_charset_name(nls), charset))
 			break;
 		if (nls->alias && !strcmp(nls->alias, charset))
 			break;
fs/ntfs/inode.c +1 -1
index bd3221cbdd95..872ef265b117 100644
@@ -2313,7 +2313,7 @@ int ntfs_show_options(struct seq_file *sf, struct dentry *root)
 		seq_printf(sf, ",fmask=0%o", vol->fmask);
 		seq_printf(sf, ",dmask=0%o", vol->dmask);
 	}
-	seq_printf(sf, ",nls=%s", vol->nls_map->charset);
+	seq_printf(sf, ",nls=%s", nls_charset_name(vol->nls_map));
 	if (NVolCaseSensitive(vol))
 		seq_printf(sf, ",case_sensitive");
 	if (NVolShowSystemFiles(vol))
fs/ntfs/super.c +3 -3
index bb7159f697f2..1c68c33e9816 100644
@@ -224,7 +224,7 @@ static bool parse_options(ntfs_volume *vol, char *opt)
 				}
 				ntfs_error(vol->sb, "NLS character set %s not "
 						"found. Using previous one %s.",
-						v, old_nls->charset);
+					   v, nls_charset_name(old_nls));
 				nls_map = old_nls;
 			} else /* nls_map */ {
 				unload_nls(old_nls);
@@ -274,7 +274,7 @@ static bool parse_options(ntfs_volume *vol, char *opt)
 					"on remount.");
 			return false;
 		} /* else (!vol->nls_map) */
-		ntfs_debug("Using NLS character set %s.", nls_map->charset);
+		ntfs_debug("Using NLS character set %s.", nls_charset_name(nls_map));
 		vol->nls_map = nls_map;
 	} else /* (!nls_map) */ {
 		if (!vol->nls_map) {
@@ -285,7 +285,7 @@ static bool parse_options(ntfs_volume *vol, char *opt)
 				return false;
 			}
 			ntfs_debug("Using default NLS character set (%s).",
-					vol->nls_map->charset);
+				   nls_charset_name(vol->nls_map));
 		}
 	}
 	if (mft_zone_multiplier != -1) {
fs/ntfs/unistr.c +3 -2
index e0a5f33441df..a30911979a55 100644
@@ -297,7 +297,7 @@ int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,
 	if (wc_len < 0) {
 		ntfs_error(vol->sb, "Name using character set %s contains "
 				"characters that cannot be converted to "
-				"Unicode.", nls->charset);
+				"Unicode.", nls_charset_name(nls));
 		i = -EILSEQ;
 	} else /* if (o >= NTFS_MAX_NAME_LEN) */ {
 		ntfs_error(vol->sb, "Name is too long (maximum length for a "
@@ -386,7 +386,8 @@ retry:			wc = nls_uni2char(nls, le16_to_cpu(ins[i]),
 conversion_err:
 	ntfs_error(vol->sb, "Unicode name contains characters that cannot be "
 			"converted to character set %s.  You might want to "
-			"try to use the mount option nls=utf8.", nls->charset);
+			"try to use the mount option nls=utf8.",
+			nls_charset_name(nls));
 	if (ns != *outs)
 		kfree(ns);
 	if (wc != -ENAMETOOLONG)
fs/udf/super.c +2 -1
index 8f2f56d9a1bb..284087eb64d0 100644
@@ -361,7 +361,8 @@ static int udf_show_options(struct seq_file *seq, struct dentry *root)
 	if (UDF_QUERY_FLAG(sb, UDF_FLAG_UTF8))
 		seq_puts(seq, ",utf8");
 	if (UDF_QUERY_FLAG(sb, UDF_FLAG_NLS_MAP) && sbi->s_nls_map)
-		seq_printf(seq, ",iocharset=%s", sbi->s_nls_map->charset);
+		seq_printf(seq, ",iocharset=%s",
+			   nls_charset_name(sbi->s_nls_map));
 
 	return 0;
 }
include/linux/nls.h +5 -0
index 5073ecd57279..cacbcd7d63e6 100644
@@ -72,6 +72,11 @@ static inline int nls_char2uni(const struct nls_table *table,
 	return table->char2uni(rawstring, boundlen, uni);
 }
 
+static inline const char *nls_charset_name(const struct nls_table *table)
+{
+	return table->charset;
+}
+
 static inline unsigned char nls_tolower(struct nls_table *t, unsigned char c)
 {
 	unsigned char nc = t->charset2lower[c];
Signature · 1 line
2.20.0.rc2
#3 lore raw

PATCH v4 3/23 nls: Wrap charset hooks in ops structure

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

This is done in preparation to splitting the nls_table structure, in order to support multiple versions of the same encoding. By placing all the operations together, we can allow multiple operations for the same encoding, depending on the version. For now, there is no behavior change intended, but this simplify the following patches.

With the exception of the declaration of the structure, this patch was generated by the following Coccinelle script:

<smpl>

@nlstable@
identifier p;
expression uni2char_fn;
expression char2uni_fn;

@@
static struct nls_table p = {

-       .char2uni = char2uni_fn,
-       .uni2char = uni2char_fn,
+       .ops = &charset_ops,

};

@createops@
identifier nlstable.p;
expression nlstable.uni2char_fn;
expression nlstable.char2uni_fn;
@@

+static const struct nls_ops charset_ops = {
+ .uni2char = uni2char_fn,
+ .char2uni = char2uni_fn,
+};
+
static struct nls_table p = {};

@@
struct nls_table *c;
@@
(
- c->uni2char
+ c->ops->uni2char
|
- c->char2uni
+ c->ops->char2uni
)

</smpl>

---
 fs/nls/mac-celtic.c     |  8 ++++++--
 fs/nls/mac-centeuro.c   |  8 ++++++--
 fs/nls/mac-croatian.c   |  8 ++++++--
 fs/nls/mac-cyrillic.c   |  8 ++++++--
 fs/nls/mac-gaelic.c     |  8 ++++++--
 fs/nls/mac-greek.c      |  8 ++++++--
 fs/nls/mac-iceland.c    |  8 ++++++--
 fs/nls/mac-inuit.c      |  8 ++++++--
 fs/nls/mac-roman.c      |  8 ++++++--
 fs/nls/mac-romanian.c   |  8 ++++++--
 fs/nls/mac-turkish.c    |  8 ++++++--
 fs/nls/nls_ascii.c      |  8 ++++++--
 fs/nls/nls_base.c       |  8 ++++++--
 fs/nls/nls_cp1250.c     |  8 ++++++--
 fs/nls/nls_cp1251.c     |  8 ++++++--
 fs/nls/nls_cp1255.c     |  8 ++++++--
 fs/nls/nls_cp437.c      |  8 ++++++--
 fs/nls/nls_cp737.c      |  8 ++++++--
 fs/nls/nls_cp775.c      |  8 ++++++--
 fs/nls/nls_cp850.c      |  8 ++++++--
 fs/nls/nls_cp852.c      |  8 ++++++--
 fs/nls/nls_cp855.c      |  8 ++++++--
 fs/nls/nls_cp857.c      |  8 ++++++--
 fs/nls/nls_cp860.c      |  8 ++++++--
 fs/nls/nls_cp861.c      |  8 ++++++--
 fs/nls/nls_cp862.c      |  8 ++++++--
 fs/nls/nls_cp863.c      |  8 ++++++--
 fs/nls/nls_cp864.c      |  8 ++++++--
 fs/nls/nls_cp865.c      |  8 ++++++--
 fs/nls/nls_cp866.c      |  8 ++++++--
 fs/nls/nls_cp869.c      |  8 ++++++--
 fs/nls/nls_cp874.c      |  8 ++++++--
 fs/nls/nls_cp932.c      |  8 ++++++--
 fs/nls/nls_cp936.c      |  8 ++++++--
 fs/nls/nls_cp949.c      |  8 ++++++--
 fs/nls/nls_cp950.c      |  8 ++++++--
 fs/nls/nls_euc-jp.c     |  8 ++++++--
 fs/nls/nls_iso8859-1.c  |  8 ++++++--
 fs/nls/nls_iso8859-13.c |  8 ++++++--
 fs/nls/nls_iso8859-14.c |  8 ++++++--
 fs/nls/nls_iso8859-15.c |  8 ++++++--
 fs/nls/nls_iso8859-2.c  |  8 ++++++--
 fs/nls/nls_iso8859-3.c  |  8 ++++++--
 fs/nls/nls_iso8859-4.c  |  8 ++++++--
 fs/nls/nls_iso8859-5.c  |  8 ++++++--
 fs/nls/nls_iso8859-6.c  |  8 ++++++--
 fs/nls/nls_iso8859-7.c  |  8 ++++++--
 fs/nls/nls_iso8859-9.c  |  8 ++++++--
 fs/nls/nls_koi8-r.c     |  8 ++++++--
 fs/nls/nls_koi8-ru.c    |  8 ++++++--
 fs/nls/nls_koi8-u.c     |  8 ++++++--
 fs/nls/nls_utf8.c       |  8 ++++++--
 fs/udf/unicode.c        |  4 ++--
 include/linux/nls.h     | 16 ++++++++++------
 54 files changed, 324 insertions(+), 112 deletions(-)
fs/nls/mac-celtic.c +6 -2
index 266c2d7d50bd..1b59b04f26f2 100644
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macceltic",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-centeuro.c +6 -2
index 9789c6057551..d5b8f38f97b6 100644
@@ -507,10 +507,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "maccenteuro",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-croatian.c +6 -2
index bb19e7a07d43..32de6accd526 100644
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "maccroatian",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-cyrillic.c +6 -2
index 2a7dea36acba..34d5c1c05ff1 100644
@@ -472,10 +472,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "maccyrillic",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-gaelic.c +6 -2
index 77b001653588..2aabf5213176 100644
@@ -542,10 +542,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macgaelic",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-greek.c +6 -2
index 1eccf499e2eb..df62909ef57e 100644
@@ -472,10 +472,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macgreek",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-iceland.c +6 -2
index cbd0875c6d69..8daa68b995bc 100644
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "maciceland",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-inuit.c +6 -2
index fba8357aaf03..b0799693502a 100644
@@ -507,10 +507,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macinuit",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-roman.c +6 -2
index b6a98a5208cd..ba358b864b05 100644
@@ -612,10 +612,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macroman",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-romanian.c +6 -2
index 25547f023638..7a8a7f9a0bbc 100644
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macromanian",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/mac-turkish.c +6 -2
index b5454bc7b7fa..eb3c5e53ec88 100644
@@ -577,10 +577,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "macturkish",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_ascii.c +6 -2
index a2620650d5e4..6bad3e779284 100644
@@ -142,10 +142,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "ascii",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_base.c +6 -2
index e5d083b6e2b2..0bb0acf6893f 100644
@@ -520,10 +520,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table default_table = {
 	.charset	= "default",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp1250.c +6 -2
index ace3e19d3407..08902e86fc8e 100644
@@ -323,10 +323,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
         return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp1250",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp1251.c +6 -2
index 9273ddfd08a1..2bb88c8cc5bf 100644
@@ -277,10 +277,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp1251",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp1255.c +6 -2
index 1caf5dfed85b..c6bf8d575c5b 100644
@@ -358,11 +358,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp1255",
 	.alias		= "iso8859-8",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp437.c +6 -2
index 7ddb830da3fd..0f3f8bdbb62b 100644
@@ -363,10 +363,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp437",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp737.c +6 -2
index c593f683a0cd..9383359ca25f 100644
@@ -326,10 +326,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp737",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp775.c +6 -2
index 554c863745f2..6c787b9079ed 100644
@@ -295,10 +295,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp775",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp850.c +6 -2
index 56cccd14b40b..50a57138a571 100644
@@ -291,10 +291,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp850",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp852.c +6 -2
index 7cdc05ac1d40..0cbb199f1cd5 100644
@@ -313,10 +313,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp852",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp855.c +6 -2
index 7426eea05663..530b77c86363 100644
@@ -275,10 +275,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp855",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp857.c +6 -2
index 098309733ebd..0db642ec6f45 100644
@@ -277,10 +277,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp857",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp860.c +6 -2
index 84224478e731..44a40dac26bd 100644
@@ -340,10 +340,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp860",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp861.c +6 -2
index dc873e4be092..50e08174fc48 100644
@@ -363,10 +363,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp861",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp862.c +6 -2
index d5263e3c5566..3505f3437972 100644
@@ -397,10 +397,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp862",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp863.c +6 -2
index 051c9832e36a..e3489cdc0c04 100644
@@ -357,10 +357,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp863",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp864.c +6 -2
index 97eb1273b2f7..d4185bc7f1bf 100644
@@ -383,10 +383,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp864",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp865.c +6 -2
index 111214228525..9f468944e577 100644
@@ -363,10 +363,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp865",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp866.c +6 -2
index ffdcbc3fc38d..ee46fd5a76b1 100644
@@ -281,10 +281,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp866",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp869.c +6 -2
index 3b5a34589354..da29a4a53e1d 100644
@@ -291,10 +291,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp869",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp874.c +6 -2
index 8dfaa10710fa..642659b9ed89 100644
@@ -249,11 +249,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp874",
 	.alias		= "tis-620",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp932.c +6 -2
index 67b7398e8483..3e7bdefdca90 100644
@@ -7907,11 +7907,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 		return -EINVAL;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp932",
 	.alias		= "sjis",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp936.c +6 -2
index c96546cfec9f..b1fa2918992b 100644
@@ -11085,11 +11085,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp936",
 	.alias		= "gb2312",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp949.c +6 -2
index 199171e97aa4..1d334095d86c 100644
@@ -13920,11 +13920,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp949",
 	.alias		= "euc-kr",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_cp950.c +6 -2
index 8e1418708209..d936160a48f9 100644
@@ -9456,11 +9456,15 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "cp950",
 	.alias		= "big5",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_euc-jp.c +6 -2
index eec257545f04..0af73982738b 100644
@@ -549,10 +549,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return euc_offset;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "euc-jp",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 };
 
 static int __init init_nls_euc_jp(void)
fs/nls/nls_iso8859-1.c +6 -2
index 69ac020d43b1..6212b2925fa0 100644
@@ -233,10 +233,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-1",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-13.c +6 -2
index afb3f8f275f0..8f0a23109207 100644
@@ -261,10 +261,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-13",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-14.c +6 -2
index 046370f0b6f0..80ab77f37480 100644
@@ -317,10 +317,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-14",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-15.c +6 -2
index 7e34a841a056..5c02f93e7b20 100644
@@ -283,10 +283,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-15",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-2.c +6 -2
index 7dd571181741..97afc1233da1 100644
@@ -284,10 +284,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-2",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-3.c +6 -2
index 740b75ec4493..f835fcec3aae 100644
@@ -284,10 +284,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-3",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-4.c +6 -2
index 8826021e32f5..14acb68fb013 100644
@@ -284,10 +284,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-4",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-5.c +6 -2
index 7c04057a1ad8..f559bbb25045 100644
@@ -248,10 +248,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-5",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-6.c +6 -2
index d4a881400d74..e3d7e28363b8 100644
@@ -239,10 +239,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-6",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-7.c +6 -2
index 37b75d825a75..49fd2b24e492 100644
@@ -293,10 +293,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-7",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_iso8859-9.c +6 -2
index 557b98250d37..876696f89626 100644
@@ -248,10 +248,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "iso8859-9",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_koi8-r.c +6 -2
index 811f232fccfb..6a85211402a8 100644
@@ -299,10 +299,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "koi8-r",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_koi8-ru.c +6 -2
index 32781252110d..c4e382fd0f13 100644
@@ -51,10 +51,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "koi8-ru",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 };
 
 static int __init init_nls_koi8_ru(void)
fs/nls/nls_koi8-u.c +6 -2
index 7e029e4c188a..5f91e9cdb165 100644
@@ -306,10 +306,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "koi8-u",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
fs/nls/nls_utf8.c +6 -2
index afcfbc4a14db..6988fffd5cf6 100644
@@ -40,10 +40,14 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return n;
 }
 
+static const struct nls_ops charset_ops = {
+	.uni2char = uni2char,
+	.char2uni = char2uni,
+};
+
 static struct nls_table table = {
 	.charset	= "utf8",
-	.uni2char	= uni2char,
-	.char2uni	= char2uni,
+	.ops = &charset_ops,
 	.charset2lower	= identity,	/* no conversion */
 	.charset2upper	= identity,
 };
fs/udf/unicode.c +2 -2
index 45234791fec2..f1a9625ade43 100644
@@ -178,7 +178,7 @@ static int udf_name_from_CS0(struct super_block *sb,
 	}
 
 	if (UDF_QUERY_FLAG(sb, UDF_FLAG_NLS_MAP))
-		conv_f = UDF_SB(sb)->s_nls_map->uni2char;
+		conv_f = UDF_SB(sb)->s_nls_map->ops->uni2char;
 	else
 		conv_f = NULL;
 
@@ -286,7 +286,7 @@ static int udf_name_to_CS0(struct super_block *sb,
 		return 0;
 
 	if (UDF_QUERY_FLAG(sb, UDF_FLAG_NLS_MAP))
-		conv_f = UDF_SB(sb)->s_nls_map->char2uni;
+		conv_f = UDF_SB(sb)->s_nls_map->ops->char2uni;
 	else
 		conv_f = NULL;
 
include/linux/nls.h +10 -6
index cacbcd7d63e6..5d63fe6aa55e 100644
@@ -22,12 +22,16 @@ typedef u16 wchar_t;
 /* Arbitrary Unicode character */
 typedef u32 unicode_t;
 
-struct nls_table {
-	const char *charset;
-	const char *alias;
+struct nls_ops {
 	int (*uni2char) (wchar_t uni, unsigned char *out, int boundlen);
 	int (*char2uni) (const unsigned char *rawstring, int boundlen,
 			 wchar_t *uni);
+};
+
+struct nls_table {
+	const char *charset;
+	const char *alias;
+	const struct nls_ops *ops;
 	const unsigned char *charset2lower;
 	const unsigned char *charset2upper;
 	struct module *owner;
@@ -62,14 +66,14 @@ extern int utf16s_to_utf8s(const wchar_t *pwcs, int len,
 static inline int nls_uni2char(const struct nls_table *table, wchar_t uni,
 			       unsigned char *out, int boundlen)
 {
-	return table->uni2char(uni, out, boundlen);
+	return table->ops->uni2char(uni, out, boundlen);
 }
 
 static inline int nls_char2uni(const struct nls_table *table,
 			       const unsigned char *rawstring,
 			       int boundlen, wchar_t *uni)
 {
-	return table->char2uni(rawstring, boundlen, uni);
+	return table->ops->char2uni(rawstring, boundlen, uni);
 }
 
 static inline const char *nls_charset_name(const struct nls_table *table)
@@ -116,7 +120,7 @@ nls_nullsize(const struct nls_table *codepage)
 	int charlen;
 	char tmp[NLS_MAX_CHARSET_SIZE];
 
-	charlen = codepage->uni2char(0, tmp, NLS_MAX_CHARSET_SIZE);
+	charlen = codepage->ops->uni2char(0, tmp, NLS_MAX_CHARSET_SIZE);
 
 	return charlen > 0 ? charlen : 1;
 }
Signature · 1 line
2.20.0.rc2
#4 lore raw

PATCH v4 4/23 nls: Split default charset from NLS core

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

---
 fs/nls/Makefile                      |  1 +
 fs/nls/nls_core.c                    | 94 ++++++++++++++++++++++++++++
 fs/nls/{nls_base.c => nls_default.c} | 93 +++------------------------
 3 files changed, 102 insertions(+), 86 deletions(-)
 create mode 100644 fs/nls/nls_core.c
 rename fs/nls/{nls_base.c => nls_default.c} (90%)
fs/nls/Makefile +1 -0
index ac54db297128..5f42ceff9d15 100644
@@ -3,6 +3,7 @@
 # Makefile for native language support
 #
 
+nls_base-y := nls_core.o nls_default.o
 obj-$(CONFIG_NLS)		+= nls_base.o
 
 obj-$(CONFIG_NLS_CODEPAGE_437)	+= nls_cp437.o
fs/nls/nls_core.c new file +94 -0

94 lines not on this page · raw message

fs/nls/nls_default.c renamed from fs/nls/nls_base.c +7 -86

118 lines not on this page · raw message

Signature · 1 line
2.20.0.rc2
#5 lore raw

PATCH v4 5/23 nls: Split struct nls_charset from struct nls_table

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

In order to support multiple versions of the same encoding, we want to split the charset registering data, which enrolls it with the NLS subsystem and is valid for all versions of the encoding, from the version specific data, that is actually going to be returned when the caller loads an NLS charset.

With the exception of the following files (files declaration and default charset), which were edited by hand, the other files were generated with the following Coccinelle patch:

Files edited by hand:
- fs/nls/nls_base.c
- include/linux/nls.h
- fs/nls/nls_default.c

<smpl>
@nlstable@
identifier p;
expression charset_str;
@@

static struct nls_table p = {
- .charset = charset_str,
};

@createops@
identifier nlstable.p;
expression nlstable.charset_str;

@@

+static struct nls_charset nls_charset;

static struct nls_table p = {
+ .charset = &nls_charset,
};
+
+static struct nls_charset nls_charset = {
+ .charset = charset_str,
+ .tables = &p,
+};

@@
expression A;
@@
? return
- register_nls(A);
+ register_nls(&nls_charset);

@@
expression A;
@@

- unregister_nls(A);
+ unregister_nls(&nls_charset);

@mvalias@
identifier p;
expression alias_str;
@@
static struct nls_table p = {
- .alias = alias_str,
};

@@
expression mvalias.alias_str;
@@
static struct nls_charset nls_charset = {
+ .alias = alias_str,
};

</smpl>

---
 fs/nls/mac-celtic.c     | 12 ++++++++---
 fs/nls/mac-centeuro.c   | 12 ++++++++---
 fs/nls/mac-croatian.c   | 12 ++++++++---
 fs/nls/mac-cyrillic.c   | 12 ++++++++---
 fs/nls/mac-gaelic.c     | 12 ++++++++---
 fs/nls/mac-greek.c      | 12 ++++++++---
 fs/nls/mac-iceland.c    | 12 ++++++++---
 fs/nls/mac-inuit.c      | 12 ++++++++---
 fs/nls/mac-roman.c      | 12 ++++++++---
 fs/nls/mac-romanian.c   | 12 ++++++++---
 fs/nls/mac-turkish.c    | 12 ++++++++---
 fs/nls/nls_ascii.c      | 12 ++++++++---
 fs/nls/nls_core.c       | 48 +++++++++++++++++++++++++++--------------
 fs/nls/nls_cp1250.c     | 12 ++++++++---
 fs/nls/nls_cp1251.c     | 12 ++++++++---
 fs/nls/nls_cp1255.c     | 14 ++++++++----
 fs/nls/nls_cp437.c      | 12 ++++++++---
 fs/nls/nls_cp737.c      | 12 ++++++++---
 fs/nls/nls_cp775.c      | 12 ++++++++---
 fs/nls/nls_cp850.c      | 12 ++++++++---
 fs/nls/nls_cp852.c      | 12 ++++++++---
 fs/nls/nls_cp855.c      | 12 ++++++++---
 fs/nls/nls_cp857.c      | 12 ++++++++---
 fs/nls/nls_cp860.c      | 12 ++++++++---
 fs/nls/nls_cp861.c      | 12 ++++++++---
 fs/nls/nls_cp862.c      | 12 ++++++++---
 fs/nls/nls_cp863.c      | 12 ++++++++---
 fs/nls/nls_cp864.c      | 12 ++++++++---
 fs/nls/nls_cp865.c      | 12 ++++++++---
 fs/nls/nls_cp866.c      | 12 ++++++++---
 fs/nls/nls_cp869.c      | 12 ++++++++---
 fs/nls/nls_cp874.c      | 14 ++++++++----
 fs/nls/nls_cp932.c      | 14 ++++++++----
 fs/nls/nls_cp936.c      | 14 ++++++++----
 fs/nls/nls_cp949.c      | 14 ++++++++----
 fs/nls/nls_cp950.c      | 14 ++++++++----
 fs/nls/nls_default.c    |  9 ++++++--
 fs/nls/nls_euc-jp.c     | 12 ++++++++---
 fs/nls/nls_iso8859-1.c  | 12 ++++++++---
 fs/nls/nls_iso8859-13.c | 12 ++++++++---
 fs/nls/nls_iso8859-14.c | 12 ++++++++---
 fs/nls/nls_iso8859-15.c | 12 ++++++++---
 fs/nls/nls_iso8859-2.c  | 12 ++++++++---
 fs/nls/nls_iso8859-3.c  | 12 ++++++++---
 fs/nls/nls_iso8859-4.c  | 12 ++++++++---
 fs/nls/nls_iso8859-5.c  | 12 ++++++++---
 fs/nls/nls_iso8859-6.c  | 12 ++++++++---
 fs/nls/nls_iso8859-7.c  | 12 ++++++++---
 fs/nls/nls_iso8859-9.c  | 12 ++++++++---
 fs/nls/nls_koi8-r.c     | 12 ++++++++---
 fs/nls/nls_koi8-ru.c    | 12 ++++++++---
 fs/nls/nls_koi8-u.c     | 12 ++++++++---
 fs/nls/nls_utf8.c       | 12 ++++++++---
 include/linux/nls.h     | 18 ++++++++++------
 54 files changed, 516 insertions(+), 183 deletions(-)
fs/nls/mac-celtic.c +9 -3
index 1b59b04f26f2..4fe7347c55d6 100644
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macceltic",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macceltic",
+	.tables = &table,
+};
+
 static int __init init_nls_macceltic(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macceltic(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macceltic)
fs/nls/mac-centeuro.c +9 -3
index d5b8f38f97b6..2d115aae4240 100644
@@ -512,21 +512,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "maccenteuro",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "maccenteuro",
+	.tables = &table,
+};
+
 static int __init init_nls_maccenteuro(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_maccenteuro(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_maccenteuro)
fs/nls/mac-croatian.c +9 -3
index 32de6accd526..b496b85fcde1 100644
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "maccroatian",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "maccroatian",
+	.tables = &table,
+};
+
 static int __init init_nls_maccroatian(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_maccroatian(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_maccroatian)
fs/nls/mac-cyrillic.c +9 -3
index 34d5c1c05ff1..18c9e0eb8e58 100644
@@ -477,21 +477,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "maccyrillic",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "maccyrillic",
+	.tables = &table,
+};
+
 static int __init init_nls_maccyrillic(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_maccyrillic(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_maccyrillic)
fs/nls/mac-gaelic.c +9 -3
index 2aabf5213176..8f8d6ae20f02 100644
@@ -547,21 +547,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macgaelic",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macgaelic",
+	.tables = &table,
+};
+
 static int __init init_nls_macgaelic(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macgaelic(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macgaelic)
fs/nls/mac-greek.c +9 -3
index df62909ef57e..0e2c12fe3447 100644
@@ -477,21 +477,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macgreek",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macgreek",
+	.tables = &table,
+};
+
 static int __init init_nls_macgreek(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macgreek(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macgreek)
fs/nls/mac-iceland.c +9 -3
index 8daa68b995bc..414767fa47a4 100644
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "maciceland",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "maciceland",
+	.tables = &table,
+};
+
 static int __init init_nls_maciceland(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_maciceland(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_maciceland)
fs/nls/mac-inuit.c +9 -3
index b0799693502a..0e06fd3a0c8f 100644
@@ -512,21 +512,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macinuit",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macinuit",
+	.tables = &table,
+};
+
 static int __init init_nls_macinuit(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macinuit(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macinuit)
fs/nls/mac-roman.c +9 -3
index ba358b864b05..fcfd387cfaa8 100644
@@ -617,21 +617,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macroman",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macroman",
+	.tables = &table,
+};
+
 static int __init init_nls_macroman(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macroman(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macroman)
fs/nls/mac-romanian.c +9 -3
index 7a8a7f9a0bbc..74027022a135 100644
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macromanian",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macromanian",
+	.tables = &table,
+};
+
 static int __init init_nls_macromanian(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macromanian(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macromanian)
fs/nls/mac-turkish.c +9 -3
index eb3c5e53ec88..0edc0f8b1f4d 100644
@@ -582,21 +582,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "macturkish",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "macturkish",
+	.tables = &table,
+};
+
 static int __init init_nls_macturkish(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_macturkish(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_macturkish)
fs/nls/nls_ascii.c +9 -3
index 6bad3e779284..3c3ee908d1ed 100644
@@ -147,21 +147,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "ascii",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "ascii",
+	.tables = &table,
+};
+
 static int __init init_nls_ascii(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_ascii(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_ascii)
fs/nls/nls_core.c +32 -16

92 lines not on this page · raw message

fs/nls/nls_cp1250.c +9 -3
index 08902e86fc8e..080717694405 100644
@@ -328,20 +328,26 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp1250",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp1250",
+	.tables = &table,
+};
+
 static int __init init_nls_cp1250(void)
 {
-        return register_nls(&table);
+        return register_nls(&nls_charset);
 }
 static void __exit exit_nls_cp1250(void)
 {
-        unregister_nls(&table);
+        unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp1250)
fs/nls/nls_cp1251.c +9 -3
index 2bb88c8cc5bf..2fba498ab289 100644
@@ -282,21 +282,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp1251",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp1251",
+	.tables = &table,
+};
+
 static int __init init_nls_cp1251(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp1251(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp1251)
fs/nls/nls_cp1255.c +10 -4
index c6bf8d575c5b..c268e8d8c038 100644
@@ -363,22 +363,28 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp1255",
-	.alias		= "iso8859-8",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.alias = "iso8859-8",
+	.charset = "cp1255",
+	.tables = &table,
+};
+
 static int __init init_nls_cp1255(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp1255(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp1255)
fs/nls/nls_cp437.c +9 -3
index 0f3f8bdbb62b..f24f8691e720 100644
@@ -368,21 +368,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp437",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp437",
+	.tables = &table,
+};
+
 static int __init init_nls_cp437(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp437(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp437)
fs/nls/nls_cp737.c +9 -3
index 9383359ca25f..f5a8b9e88165 100644
@@ -331,21 +331,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp737",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp737",
+	.tables = &table,
+};
+
 static int __init init_nls_cp737(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp737(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp737)
fs/nls/nls_cp775.c +9 -3
index 6c787b9079ed..d268bfb873e4 100644
@@ -300,21 +300,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp775",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp775",
+	.tables = &table,
+};
+
 static int __init init_nls_cp775(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp775(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp775)
fs/nls/nls_cp850.c +9 -3
index 50a57138a571..b698b0df65e3 100644
@@ -296,21 +296,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp850",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp850",
+	.tables = &table,
+};
+
 static int __init init_nls_cp850(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp850(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp850)
fs/nls/nls_cp852.c +9 -3
index 0cbb199f1cd5..738e95346b34 100644
@@ -318,21 +318,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp852",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp852",
+	.tables = &table,
+};
+
 static int __init init_nls_cp852(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp852(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp852)
fs/nls/nls_cp855.c +9 -3
index 530b77c86363..9a1c4e307cb1 100644
@@ -280,21 +280,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp855",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp855",
+	.tables = &table,
+};
+
 static int __init init_nls_cp855(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp855(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp855)
fs/nls/nls_cp857.c +9 -3
index 0db642ec6f45..782e31cb9f5a 100644
@@ -282,21 +282,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp857",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp857",
+	.tables = &table,
+};
+
 static int __init init_nls_cp857(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp857(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp857)
fs/nls/nls_cp860.c +9 -3
index 44a40dac26bd..2ad1954b84e6 100644
@@ -345,21 +345,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp860",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp860",
+	.tables = &table,
+};
+
 static int __init init_nls_cp860(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp860(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp860)
fs/nls/nls_cp861.c +9 -3
index 50e08174fc48..5930b0e6e8f1 100644
@@ -368,21 +368,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp861",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp861",
+	.tables = &table,
+};
+
 static int __init init_nls_cp861(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp861(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp861)
fs/nls/nls_cp862.c +9 -3
index 3505f3437972..63c27b24a011 100644
@@ -402,21 +402,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp862",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp862",
+	.tables = &table,
+};
+
 static int __init init_nls_cp862(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp862(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp862)
fs/nls/nls_cp863.c +9 -3
index e3489cdc0c04..aa815cdc7481 100644
@@ -362,21 +362,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp863",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp863",
+	.tables = &table,
+};
+
 static int __init init_nls_cp863(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp863(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp863)
fs/nls/nls_cp864.c +9 -3
index d4185bc7f1bf..a20725f661e9 100644
@@ -388,21 +388,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp864",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp864",
+	.tables = &table,
+};
+
 static int __init init_nls_cp864(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp864(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp864)
fs/nls/nls_cp865.c +9 -3
index 9f468944e577..3d22ec2bd7af 100644
@@ -368,21 +368,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp865",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp865",
+	.tables = &table,
+};
+
 static int __init init_nls_cp865(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp865(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp865)
fs/nls/nls_cp866.c +9 -3
index ee46fd5a76b1..35dc7b2f023a 100644
@@ -286,21 +286,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp866",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp866",
+	.tables = &table,
+};
+
 static int __init init_nls_cp866(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp866(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp866)
fs/nls/nls_cp869.c +9 -3
index da29a4a53e1d..56504ab0f405 100644
@@ -296,21 +296,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp869",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "cp869",
+	.tables = &table,
+};
+
 static int __init init_nls_cp869(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp869(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp869)
fs/nls/nls_cp874.c +10 -4
index 642659b9ed89..41394620d000 100644
@@ -254,22 +254,28 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp874",
-	.alias		= "tis-620",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.alias = "tis-620",
+	.charset = "cp874",
+	.tables = &table,
+};
+
 static int __init init_nls_cp874(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp874(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp874)
fs/nls/nls_cp932.c +10 -4
index 3e7bdefdca90..25fe26fb2603 100644
@@ -7912,22 +7912,28 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp932",
-	.alias		= "sjis",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.alias = "sjis",
+	.charset = "cp932",
+	.tables = &table,
+};
+
 static int __init init_nls_cp932(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp932(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp932)
fs/nls/nls_cp936.c +10 -4
index b1fa2918992b..766f86b53a7b 100644
@@ -11090,22 +11090,28 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp936",
-	.alias		= "gb2312",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.alias = "gb2312",
+	.charset = "cp936",
+	.tables = &table,
+};
+
 static int __init init_nls_cp936(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp936(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp936)
fs/nls/nls_cp949.c +10 -4
index 1d334095d86c..138eec74bb3f 100644
@@ -13925,22 +13925,28 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp949",
-	.alias		= "euc-kr",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.alias = "euc-kr",
+	.charset = "cp949",
+	.tables = &table,
+};
+
 static int __init init_nls_cp949(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp949(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp949)
fs/nls/nls_cp950.c +10 -4
index d936160a48f9..899da09fe0d7 100644
@@ -9461,22 +9461,28 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "cp950",
-	.alias		= "big5",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.alias = "big5",
+	.charset = "cp950",
+	.tables = &table,
+};
+
 static int __init init_nls_cp950(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_cp950(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_cp950)
fs/nls/nls_default.c +7 -2
index c5d7e8391b22..ef8c0efb8a3c 100644
@@ -17,7 +17,7 @@
 #include <asm/byteorder.h>
 #include <linux/nls.h>
 
-static struct nls_table default_table;
+struct nls_charset default_charset;
 
 struct utf8_table {
 	int     cmask;
@@ -453,12 +453,17 @@ static const struct nls_ops charset_ops = {
 };
 
 static struct nls_table default_table = {
-	.charset	= "default",
+	.charset = &default_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+struct nls_charset default_charset = {
+	.charset = "default",
+	.tables = &default_table,
+};
+
 /* Returns a simple default translation table */
 struct nls_table *load_nls_default(void)
 {
fs/nls/nls_euc-jp.c +9 -3
index 0af73982738b..8bc5d9991452 100644
@@ -554,11 +554,17 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "euc-jp",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "euc-jp",
+	.tables = &table,
+};
+
 static int __init init_nls_euc_jp(void)
 {
 	p_nls = load_nls("cp932");
@@ -566,7 +572,7 @@ static int __init init_nls_euc_jp(void)
 	if (p_nls) {
 		table.charset2upper = p_nls->charset2upper;
 		table.charset2lower = p_nls->charset2lower;
-		return register_nls(&table);
+		return register_nls(&nls_charset);
 	}
 
 	return -EINVAL;
@@ -574,7 +580,7 @@ static int __init init_nls_euc_jp(void)
 
 static void __exit exit_nls_euc_jp(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 	unload_nls(p_nls);
 }
 
fs/nls/nls_iso8859-1.c +9 -3
index 6212b2925fa0..78e9c0169f69 100644
@@ -238,21 +238,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-1",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-1",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_1(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_1(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_1)
fs/nls/nls_iso8859-13.c +9 -3
index 8f0a23109207..eb8665629e0f 100644
@@ -266,21 +266,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-13",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-13",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_13(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_13(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_13)
fs/nls/nls_iso8859-14.c +9 -3
index 80ab77f37480..c8d5a48f869c 100644
@@ -322,21 +322,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-14",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-14",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_14(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_14(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_14)
fs/nls/nls_iso8859-15.c +9 -3
index 5c02f93e7b20..0611c6cb56b4 100644
@@ -288,21 +288,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-15",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-15",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_15(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_15(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_15)
fs/nls/nls_iso8859-2.c +9 -3
index 97afc1233da1..5255d92a25eb 100644
@@ -289,21 +289,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-2",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-2",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_2(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_2(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_2)
fs/nls/nls_iso8859-3.c +9 -3
index f835fcec3aae..ad1b84f3e102 100644
@@ -289,21 +289,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-3",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-3",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_3(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_3(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_3)
fs/nls/nls_iso8859-4.c +9 -3
index 14acb68fb013..82469deee0ba 100644
@@ -289,21 +289,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-4",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-4",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_4(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_4(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_4)
fs/nls/nls_iso8859-5.c +9 -3
index f559bbb25045..3f3cd0c28797 100644
@@ -253,21 +253,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-5",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-5",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_5(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_5(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_5)
fs/nls/nls_iso8859-6.c +9 -3
index e3d7e28363b8..43e6675998bc 100644
@@ -244,21 +244,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-6",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-6",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_6(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_6(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_6)
fs/nls/nls_iso8859-7.c +9 -3
index 49fd2b24e492..83893e487f82 100644
@@ -298,21 +298,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-7",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-7",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_7(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_7(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_7)
fs/nls/nls_iso8859-9.c +9 -3
index 876696f89626..df03f97cd9d1 100644
@@ -253,21 +253,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "iso8859-9",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "iso8859-9",
+	.tables = &table,
+};
+
 static int __init init_nls_iso8859_9(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_iso8859_9(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_iso8859_9)
fs/nls/nls_koi8-r.c +9 -3
index 6a85211402a8..22918e154dbe 100644
@@ -304,21 +304,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "koi8-r",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "koi8-r",
+	.tables = &table,
+};
+
 static int __init init_nls_koi8_r(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_koi8_r(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_koi8_r)
fs/nls/nls_koi8-ru.c +9 -3
index c4e382fd0f13..f4edbc313706 100644
@@ -56,11 +56,17 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "koi8-ru",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "koi8-ru",
+	.tables = &table,
+};
+
 static int __init init_nls_koi8_ru(void)
 {
 	p_nls = load_nls("koi8-u");
@@ -68,7 +74,7 @@ static int __init init_nls_koi8_ru(void)
 	if (p_nls) {
 		table.charset2upper = p_nls->charset2upper;
 		table.charset2lower = p_nls->charset2lower;
-		return register_nls(&table);
+		return register_nls(&nls_charset);
 	}
 
 	return -EINVAL;
@@ -76,7 +82,7 @@ static int __init init_nls_koi8_ru(void)
 
 static void __exit exit_nls_koi8_ru(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 	unload_nls(p_nls);
 }
 
fs/nls/nls_koi8-u.c +9 -3
index 5f91e9cdb165..b2421625e98b 100644
@@ -311,21 +311,27 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "koi8-u",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= charset2lower,
 	.charset2upper	= charset2upper,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "koi8-u",
+	.tables = &table,
+};
+
 static int __init init_nls_koi8_u(void)
 {
-	return register_nls(&table);
+	return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_koi8_u(void)
 {
-	unregister_nls(&table);
+	unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_koi8_u)
fs/nls/nls_utf8.c +9 -3
index 6988fffd5cf6..aecf460827ac 100644
@@ -45,25 +45,31 @@ static const struct nls_ops charset_ops = {
 	.char2uni = char2uni,
 };
 
+static struct nls_charset nls_charset;
 static struct nls_table table = {
-	.charset	= "utf8",
+	.charset = &nls_charset,
 	.ops = &charset_ops,
 	.charset2lower	= identity,	/* no conversion */
 	.charset2upper	= identity,
 };
 
+static struct nls_charset nls_charset = {
+	.charset = "utf8",
+	.tables = &table,
+};
+
 static int __init init_nls_utf8(void)
 {
 	int i;
 	for (i=0; i<256; i++)
 		identity[i] = i;
 
-        return register_nls(&table);
+        return register_nls(&nls_charset);
 }
 
 static void __exit exit_nls_utf8(void)
 {
-        unregister_nls(&table);
+        unregister_nls(&nls_charset);
 }
 
 module_init(init_nls_utf8)
include/linux/nls.h +12 -6
index 5d63fe6aa55e..cdc95cd9e5d4 100644
@@ -29,15 +29,21 @@ struct nls_ops {
 };
 
 struct nls_table {
-	const char *charset;
-	const char *alias;
+	const struct nls_charset *charset;
 	const struct nls_ops *ops;
 	const unsigned char *charset2lower;
 	const unsigned char *charset2upper;
-	struct module *owner;
 	struct nls_table *next;
 };
 
+struct nls_charset {
+	const char *charset;
+	const char *alias;
+	struct module *owner;
+	struct nls_table *tables;
+	struct nls_charset *next;
+};
+
 /* this value hold the maximum octet of charset */
 #define NLS_MAX_CHARSET_SIZE 6 /* for UTF-8 */
 
@@ -49,8 +55,8 @@ enum utf16_endian {
 };
 
 /* nls_base.c */
-extern int __register_nls(struct nls_table *, struct module *);
-extern int unregister_nls(struct nls_table *);
+extern int __register_nls(struct nls_charset *, struct module *);
+extern int unregister_nls(struct nls_charset *);
 extern struct nls_table *load_nls(char *);
 extern void unload_nls(struct nls_table *);
 extern struct nls_table *load_nls_default(void);
@@ -78,7 +84,7 @@ static inline int nls_char2uni(const struct nls_table *table,
 
 static inline const char *nls_charset_name(const struct nls_table *table)
 {
-	return table->charset;
+	return table->charset->charset;
 }
 
 static inline unsigned char nls_tolower(struct nls_table *t, unsigned char c)
Signature · 1 line
2.20.0.rc2
#6 lore raw

PATCH v4 6/23 nls: Add support for multiple versions of an encoding

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

This allows a user to request a specific version of an encoding, like the version 10.0.0 of Unicode encoded in utf8.

Supporting specific versions of encodings is important to ensure stability of names in filesystems, specially when doing transformations like casefold and normalization. Even for unicode, where defined code-points are stable, there is instability for code points that weren't defined on a previous version, so the user might want to use an older version of the encoding to ensure the encoding is exact.

Not every NLS charset supports this feature. It doesn't make sense for many of them, like ASCII. Others just don't implement it yet, and never will. In those cases, the interface allows the caller to get the un-versioned charset, which is the same original behavior as if this patch weren't applied. A user that is not interested in a specific version can also ask for a versioned charset without specifying the version, and in this case, NLS will return the latest version available of that charset.

---
 fs/nls/nls_core.c   | 45 ++++++++++++++++++++++++++++++++++++++-------
 include/linux/nls.h |  8 ++++++++
 2 files changed, 46 insertions(+), 7 deletions(-)
fs/nls/nls_core.c +38 -7
index 200a7f8165e6..20e00a8b968c 100644
@@ -19,10 +19,26 @@
 extern struct nls_charset default_charset;
 static struct nls_charset *charsets = &default_charset;
 static DEFINE_SPINLOCK(nls_lock);
-static struct nls_table *nls_load_table(struct nls_charset *charset)
+
+static struct nls_table *nls_load_table(struct nls_charset *charset,
+					const char *version,
+					unsigned int flags)
 {
-	/* For now, return the default table, which is the first one found. */
-	return charset->tables;
+	struct nls_table *tbl;
+
+	/* If there is no load_table hook, only 1 table is supported and
+	 * it must have been loaded statically.
+	 */
+	if (charset->load_table)
+		tbl = charset->load_table(version, flags);
+	else
+		tbl = charset->tables;
+
+	if (IS_ERR(tbl))
+		return tbl;
+
+	tbl->flags = flags;
+	return tbl;
 }
 
 int __register_nls(struct nls_charset *nls, struct module *owner)
@@ -85,21 +101,36 @@ static struct nls_charset *find_nls(const char *charset)
 	return nls;
 }
 
-struct nls_table *load_nls(char *charset)
+struct nls_table *load_nls_version(const char *charset, const char *version,
+				   unsigned int flags)
 {
 	struct nls_charset *nls_charset;
 
 	nls_charset = try_then_request_module(find_nls(charset),
 					      "nls_%s", charset);
-	if (!IS_ERR(nls_charset))
+	if (IS_ERR(nls_charset))
+		return ERR_PTR(-EINVAL);
+
+	return nls_load_table(nls_charset, version, flags);
+}
+EXPORT_SYMBOL(load_nls_version);
+
+struct nls_table *load_nls(char *charset)
+{
+	struct nls_table *table = load_nls_version(charset, NULL, 0);
+
+	/* Pre-versioned load_nls() didn't return error pointers. Let's
+	 * keep the abi for now to prevent breakage.
+	 */
+	if (IS_ERR(table))
 		return NULL;
 
-	return nls_load_table(nls_charset);
+	return table;
 }
 
 void unload_nls(struct nls_table *nls)
 {
-	if (nls)
+	if (!IS_ERR_OR_NULL(nls))
 		module_put(nls->charset->owner);
 }
 
include/linux/nls.h +8 -0
index cdc95cd9e5d4..91524bb4477b 100644
@@ -30,6 +30,9 @@ struct nls_ops {
 
 struct nls_table {
 	const struct nls_charset *charset;
+	unsigned int version;
+	unsigned int flags;
+
 	const struct nls_ops *ops;
 	const unsigned char *charset2lower;
 	const unsigned char *charset2upper;
@@ -42,6 +45,8 @@ struct nls_charset {
 	struct module *owner;
 	struct nls_table *tables;
 	struct nls_charset *next;
+	struct nls_table *(*load_table)(const char *version,
+					unsigned int flags);
 };
 
 /* this value hold the maximum octet of charset */
@@ -58,6 +63,9 @@ enum utf16_endian {
 extern int __register_nls(struct nls_charset *, struct module *);
 extern int unregister_nls(struct nls_charset *);
 extern struct nls_table *load_nls(char *);
+extern struct nls_table *load_nls_version(const char *charset,
+					  const char *version,
+					  unsigned int flags);
 extern void unload_nls(struct nls_table *);
 extern struct nls_table *load_nls_default(void);
 #define register_nls(nls) __register_nls((nls), THIS_MODULE)
Signature · 1 line
2.20.0.rc2
#7 lore raw

PATCH v4 7/23 nls: Implement NLS_STRICT_MODE flag

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

The flag NLS_STRICT_MODE indicates whether NLS should reject invalid characters or ignore them. Support for this relies on the .validate() hook, which is implemented by each charset and states whether a given string is valid within that charset.

---
 fs/nls/nls_core.c   | 11 +++++++++++
 include/linux/nls.h | 25 +++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
fs/nls/nls_core.c +11 -0
index 20e00a8b968c..49a15bb2174f 100644
@@ -20,6 +20,14 @@ extern struct nls_charset default_charset;
 static struct nls_charset *charsets = &default_charset;
 static DEFINE_SPINLOCK(nls_lock);
 
+static int nls_validate_flags(struct nls_table *table, unsigned int flags)
+{
+	if (flags & NLS_STRICT_MODE && !table->ops->validate)
+		return -1;
+
+	return 0;
+}
+
 static struct nls_table *nls_load_table(struct nls_charset *charset,
 					const char *version,
 					unsigned int flags)
@@ -37,6 +45,9 @@ static struct nls_table *nls_load_table(struct nls_charset *charset,
 	if (IS_ERR(tbl))
 		return tbl;
 
+	if (nls_validate_flags(tbl, flags) < 0)
+		return ERR_PTR(-EINVAL);
+
 	tbl->flags = flags;
 	return tbl;
 }
include/linux/nls.h +25 -0
index 91524bb4477b..9f61015a54bf 100644
@@ -22,10 +22,22 @@ typedef u16 wchar_t;
 /* Arbitrary Unicode character */
 typedef u32 unicode_t;
 
+struct nls_table;
+
 struct nls_ops {
 	int (*uni2char) (wchar_t uni, unsigned char *out, int boundlen);
 	int (*char2uni) (const unsigned char *rawstring, int boundlen,
 			 wchar_t *uni);
+	/**
+	 * @validate:
+	 *
+	 * Returns 0 if the argument is a valid string in this charset.
+	 * Otherwise, return non-zero.
+	 *
+	 * This is required iff the charset supports strict mode.
+	 **/
+	int (*validate)(const struct nls_table *charset,
+			const unsigned char *str, size_t len);
 };
 
 struct nls_table {
@@ -59,6 +71,13 @@ enum utf16_endian {
 	UTF16_BIG_ENDIAN
 };
 
+#define NLS_STRICT_MODE			0x00000001
+
+static inline int IS_STRICT_MODE(const struct nls_table *charset)
+{
+	return (charset->flags & NLS_STRICT_MODE);
+}
+
 /* nls_base.c */
 extern int __register_nls(struct nls_charset *, struct module *);
 extern int unregister_nls(struct nls_charset *);
@@ -90,6 +109,12 @@ static inline int nls_char2uni(const struct nls_table *table,
 	return table->ops->char2uni(rawstring, boundlen, uni);
 }
 
+static inline int nls_validate(const struct nls_table *t, const unsigned char *str,
+			       const size_t len)
+{
+	return t->ops->validate(t, str, len);
+}
+
 static inline const char *nls_charset_name(const struct nls_table *table)
 {
 	return table->charset->charset;
Signature · 1 line
2.20.0.rc2
#8 lore raw

PATCH v4 8/23 nls: Let charsets define the behavior of tolower/toupper

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

Instead of always reading from a table, give the charset a chance to implement tolower() and toupper() algorithmically.

This allow us to drop a lot of tables which hardcode the identity functions (like ASCII), and replace them with a few lines of code in the hooks.

This patch was created using the semantic patch below, with the exception of the header files (hook definitions) and a fix to files that didn't have the tables statically allocated (koi8-u and cp932).

<smpl>

@tbl@
identifier p;
expression lower_tbl;
expression upper_tbl;
@@

static struct nls_table p = {
- .charset2lower = lower_tbl,
- .charset2upper = upper_tbl,
};

@@
identifier charset_ops;
expression tbl.lower_tbl;
expression tbl.upper_tbl;
@@

+ static unsigned char charset_tolower(const struct nls_table *table, unsigned int c)
+ {
+ return lower_tbl[c];
+ }
+
+ static unsigned char charset_toupper(const struct nls_table *table, unsigned int c)
+ {
+ return upper_tbl[c];
+ }

static const struct nls_ops charset_ops = {
+ .lowercase = charset_toupper,
+ .uppercase = charset_tolower,
};

@@
struct nls_table *t;
expression A;
expression nc;
@@

(
- nc = t->charset2lower[A]
+ nc = nls_tolower(t, A)

|
- nc = t->charset2upper[A]
+ nc = nls_toupper(t, A)
)
<...
- if(!nc)
- nc = A;
...>

</smpl>

---
 fs/fat/dir.c            |  5 +----
 fs/nls/mac-celtic.c     | 14 ++++++++++++--
 fs/nls/mac-centeuro.c   | 14 ++++++++++++--
 fs/nls/mac-croatian.c   | 14 ++++++++++++--
 fs/nls/mac-cyrillic.c   | 14 ++++++++++++--
 fs/nls/mac-gaelic.c     | 14 ++++++++++++--
 fs/nls/mac-greek.c      | 14 ++++++++++++--
 fs/nls/mac-iceland.c    | 14 ++++++++++++--
 fs/nls/mac-inuit.c      | 14 ++++++++++++--
 fs/nls/mac-roman.c      | 14 ++++++++++++--
 fs/nls/mac-romanian.c   | 14 ++++++++++++--
 fs/nls/mac-turkish.c    | 14 ++++++++++++--
 fs/nls/nls_ascii.c      | 14 ++++++++++++--
 fs/nls/nls_cp1250.c     | 14 ++++++++++++--
 fs/nls/nls_cp1251.c     | 14 ++++++++++++--
 fs/nls/nls_cp1255.c     | 14 ++++++++++++--
 fs/nls/nls_cp437.c      | 14 ++++++++++++--
 fs/nls/nls_cp737.c      | 14 ++++++++++++--
 fs/nls/nls_cp775.c      | 14 ++++++++++++--
 fs/nls/nls_cp850.c      | 14 ++++++++++++--
 fs/nls/nls_cp852.c      | 14 ++++++++++++--
 fs/nls/nls_cp855.c      | 14 ++++++++++++--
 fs/nls/nls_cp857.c      | 14 ++++++++++++--
 fs/nls/nls_cp860.c      | 14 ++++++++++++--
 fs/nls/nls_cp861.c      | 14 ++++++++++++--
 fs/nls/nls_cp862.c      | 14 ++++++++++++--
 fs/nls/nls_cp863.c      | 14 ++++++++++++--
 fs/nls/nls_cp864.c      | 14 ++++++++++++--
 fs/nls/nls_cp865.c      | 14 ++++++++++++--
 fs/nls/nls_cp866.c      | 14 ++++++++++++--
 fs/nls/nls_cp869.c      | 14 ++++++++++++--
 fs/nls/nls_cp874.c      | 14 ++++++++++++--
 fs/nls/nls_cp932.c      | 14 ++++++++++++--
 fs/nls/nls_cp936.c      | 14 ++++++++++++--
 fs/nls/nls_cp949.c      | 14 ++++++++++++--
 fs/nls/nls_cp950.c      | 14 ++++++++++++--
 fs/nls/nls_default.c    | 14 ++++++++++++--
 fs/nls/nls_euc-jp.c     |  7 ++++---
 fs/nls/nls_iso8859-1.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-13.c | 14 ++++++++++++--
 fs/nls/nls_iso8859-14.c | 14 ++++++++++++--
 fs/nls/nls_iso8859-15.c | 14 ++++++++++++--
 fs/nls/nls_iso8859-2.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-3.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-4.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-5.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-6.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-7.c  | 14 ++++++++++++--
 fs/nls/nls_iso8859-9.c  | 14 ++++++++++++--
 fs/nls/nls_koi8-r.c     | 14 ++++++++++++--
 fs/nls/nls_koi8-ru.c    |  6 +++---
 fs/nls/nls_koi8-u.c     | 14 ++++++++++++--
 fs/nls/nls_utf8.c       | 14 ++++++++++++--
 include/linux/nls.h     | 17 +++++++++++------
 54 files changed, 619 insertions(+), 116 deletions(-)
fs/fat/dir.c +1 -4
index d5f856651a08..6518886ee5cf 100644
@@ -215,10 +215,7 @@ fat_short2lower_uni(struct nls_table *t, unsigned char *c,
 		*uni = 0x003f;	/* a question mark */
 		charlen = 1;
 	} else if (charlen <= 1) {
-		unsigned char nc = t->charset2lower[*c];
-
-		if (!nc)
-			nc = *c;
+		unsigned char nc = nls_tolower(t, *c);
 
 		charlen = nls_char2uni(t, &nc, 1, uni);
 		if (charlen < 0) {
fs/nls/mac-celtic.c +12 -2
index 4fe7347c55d6..7207f9a14342 100644
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-centeuro.c +12 -2
index 2d115aae4240..0664408e4451 100644
@@ -507,7 +507,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -516,8 +528,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-croatian.c +12 -2
index b496b85fcde1..a4b7992ef8ec 100644
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-cyrillic.c +12 -2
index 18c9e0eb8e58..cb60563911ea 100644
@@ -472,7 +472,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -481,8 +493,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-gaelic.c +12 -2
index 8f8d6ae20f02..e683881f4a13 100644
@@ -542,7 +542,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -551,8 +563,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-greek.c +12 -2
index 0e2c12fe3447..bd2245238512 100644
@@ -472,7 +472,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -481,8 +493,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-iceland.c +12 -2
index 414767fa47a4..3ce3e27b3660 100644
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-inuit.c +12 -2
index 0e06fd3a0c8f..6f12cccccb37 100644
@@ -507,7 +507,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -516,8 +528,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-roman.c +12 -2
index fcfd387cfaa8..d8e411c82c69 100644
@@ -612,7 +612,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -621,8 +633,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-romanian.c +12 -2
index 74027022a135..cd638dfe9d7c 100644
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/mac-turkish.c +12 -2
index 0edc0f8b1f4d..82ba6f6b4c24 100644
@@ -577,7 +577,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -586,8 +598,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_ascii.c +12 -2
index 3c3ee908d1ed..2f4826478d3d 100644
@@ -142,7 +142,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -151,8 +163,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp1250.c +12 -2
index 080717694405..1cfe65851185 100644
@@ -323,7 +323,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
         return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -332,8 +344,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp1251.c +12 -2
index 2fba498ab289..061eb23892f1 100644
@@ -277,7 +277,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -286,8 +298,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp1255.c +12 -2
index c268e8d8c038..2a71dc175c9b 100644
@@ -358,7 +358,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -367,8 +379,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp437.c +12 -2
index f24f8691e720..4f763761b699 100644
@@ -363,7 +363,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -372,8 +384,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp737.c +12 -2
index f5a8b9e88165..2f2ab91340e7 100644
@@ -326,7 +326,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -335,8 +347,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp775.c +12 -2
index d268bfb873e4..92f311e620f3 100644
@@ -295,7 +295,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -304,8 +316,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp850.c +12 -2
index b698b0df65e3..77cdce20ced6 100644
@@ -291,7 +291,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -300,8 +312,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp852.c +12 -2
index 738e95346b34..47722904e9f1 100644
@@ -313,7 +313,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -322,8 +334,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp855.c +12 -2
index 9a1c4e307cb1..b52709886900 100644
@@ -275,7 +275,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -284,8 +296,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp857.c +12 -2
index 782e31cb9f5a..fcdf30a540f8 100644
@@ -277,7 +277,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -286,8 +298,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp860.c +12 -2
index 2ad1954b84e6..a1504424e923 100644
@@ -340,7 +340,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -349,8 +361,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp861.c +12 -2
index 5930b0e6e8f1..9fa1f54cee0d 100644
@@ -363,7 +363,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -372,8 +384,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp862.c +12 -2
index 63c27b24a011..00474e2b2102 100644
@@ -397,7 +397,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -406,8 +418,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp863.c +12 -2
index aa815cdc7481..908e573c1c42 100644
@@ -357,7 +357,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -366,8 +378,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp864.c +12 -2
index a20725f661e9..6cae9e9c73aa 100644
@@ -383,7 +383,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -392,8 +404,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp865.c +12 -2
index 3d22ec2bd7af..5aa6415ec357 100644
@@ -363,7 +363,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -372,8 +384,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp866.c +12 -2
index 35dc7b2f023a..f24b73839680 100644
@@ -281,7 +281,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -290,8 +302,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp869.c +12 -2
index 56504ab0f405..c2ba80140906 100644
@@ -291,7 +291,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -300,8 +312,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp874.c +12 -2
index 41394620d000..844bb205deee 100644
@@ -249,7 +249,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -258,8 +270,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp932.c +12 -2
index 25fe26fb2603..0a5db2a0a6b3 100644
@@ -7907,7 +7907,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 		return -EINVAL;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -7916,8 +7928,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp936.c +12 -2
index 766f86b53a7b..6b0d725cdfab 100644
@@ -11085,7 +11085,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -11094,8 +11106,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp949.c +12 -2
index 138eec74bb3f..292c2d02d2c2 100644
@@ -13920,7 +13920,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -13929,8 +13941,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_cp950.c +12 -2
index 899da09fe0d7..d4e35bfd8dbd 100644
@@ -9456,7 +9456,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -9465,8 +9477,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_default.c +12 -2
index ef8c0efb8a3c..602eeec24b3d 100644
@@ -447,7 +447,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -455,8 +467,6 @@ static const struct nls_ops charset_ops = {
 static struct nls_table default_table = {
 	.charset = &default_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 struct nls_charset default_charset = {
fs/nls/nls_euc-jp.c +4 -3
index 8bc5d9991452..b3a81350cbea 100644
@@ -549,7 +549,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return euc_offset;
 }
 
-static const struct nls_ops charset_ops = {
+static struct nls_ops charset_ops = {
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -570,8 +570,9 @@ static int __init init_nls_euc_jp(void)
 	p_nls = load_nls("cp932");
 
 	if (p_nls) {
-		table.charset2upper = p_nls->charset2upper;
-		table.charset2lower = p_nls->charset2lower;
+
+		charset_ops.uppercase = p_nls->ops->uppercase;
+		charset_ops.lowercase = p_nls->ops->lowercase;
 		return register_nls(&nls_charset);
 	}
 
fs/nls/nls_iso8859-1.c +12 -2
index 78e9c0169f69..a98298bd5de5 100644
@@ -233,7 +233,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -242,8 +254,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-13.c +12 -2
index eb8665629e0f..811f4cf1d1a3 100644
@@ -261,7 +261,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -270,8 +282,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-14.c +12 -2
index c8d5a48f869c..d8dafca31d26 100644
@@ -317,7 +317,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -326,8 +338,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-15.c +12 -2
index 0611c6cb56b4..9de12c9e25a3 100644
@@ -283,7 +283,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -292,8 +304,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-2.c +12 -2
index 5255d92a25eb..c59e2424f2b5 100644
@@ -284,7 +284,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -293,8 +305,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-3.c +12 -2
index ad1b84f3e102..4bab1b607059 100644
@@ -284,7 +284,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -293,8 +305,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-4.c +12 -2
index 82469deee0ba..1a3cf5f507f6 100644
@@ -284,7 +284,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -293,8 +305,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-5.c +12 -2
index 3f3cd0c28797..0a26cea9d578 100644
@@ -248,7 +248,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -257,8 +269,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-6.c +12 -2
index 43e6675998bc..d5a230888eed 100644
@@ -239,7 +239,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -248,8 +260,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-7.c +12 -2
index 83893e487f82..a5a171849ae4 100644
@@ -293,7 +293,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -302,8 +314,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_iso8859-9.c +12 -2
index df03f97cd9d1..795093547cd6 100644
@@ -248,7 +248,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -257,8 +269,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_koi8-r.c +12 -2
index 22918e154dbe..bbce9a608419 100644
@@ -299,7 +299,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -308,8 +320,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_koi8-ru.c +3 -3
index f4edbc313706..d3e946652bf6 100644
@@ -51,7 +51,7 @@ static int char2uni(const unsigned char *rawstring, int boundlen,
 	return n;
 }
 
-static const struct nls_ops charset_ops = {
+static struct nls_ops charset_ops = {
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -72,8 +72,8 @@ static int __init init_nls_koi8_ru(void)
 	p_nls = load_nls("koi8-u");
 
 	if (p_nls) {
-		table.charset2upper = p_nls->charset2upper;
-		table.charset2lower = p_nls->charset2lower;
+		charset_ops.uppercase = p_nls->ops->uppercase;
+		charset_ops.lowercase = p_nls->ops->lowercase;
 		return register_nls(&nls_charset);
 	}
 
fs/nls/nls_koi8-u.c +12 -2
index b2421625e98b..5de52a74f0b3 100644
@@ -306,7 +306,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return charset2lower[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return charset2upper[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -315,8 +327,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= charset2lower,
-	.charset2upper	= charset2upper,
 };
 
 static struct nls_charset nls_charset = {
fs/nls/nls_utf8.c +12 -2
index aecf460827ac..fe1ac5efaa37 100644
@@ -40,7 +40,19 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return n;
 }
 
+static unsigned char charset_tolower(const struct nls_table *table,
+				     unsigned int c){
+	return identity[c];
+}
+
+static unsigned char charset_toupper(const struct nls_table *table,
+				     unsigned int c) {
+	return identity[c];
+}
+
 static const struct nls_ops charset_ops = {
+	.lowercase = charset_toupper,
+	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
 };
@@ -49,8 +61,6 @@ static struct nls_charset nls_charset;
 static struct nls_table table = {
 	.charset = &nls_charset,
 	.ops = &charset_ops,
-	.charset2lower	= identity,	/* no conversion */
-	.charset2upper	= identity,
 };
 
 static struct nls_charset nls_charset = {
include/linux/nls.h +11 -6
index 9f61015a54bf..c43746bd390e 100644
@@ -38,6 +38,10 @@ struct nls_ops {
 	 **/
 	int (*validate)(const struct nls_table *charset,
 			const unsigned char *str, size_t len);
+	unsigned char (*lowercase)(const struct nls_table *charset,
+				   unsigned int c);
+	unsigned char (*uppercase)(const struct nls_table *charset,
+				   unsigned int c);
 };
 
 struct nls_table {
@@ -46,9 +50,8 @@ struct nls_table {
 	unsigned int flags;
 
 	const struct nls_ops *ops;
-	const unsigned char *charset2lower;
-	const unsigned char *charset2upper;
 	struct nls_table *next;
+
 };
 
 struct nls_charset {
@@ -120,16 +123,18 @@ static inline const char *nls_charset_name(const struct nls_table *table)
 	return table->charset->charset;
 }
 
-static inline unsigned char nls_tolower(struct nls_table *t, unsigned char c)
+static inline unsigned char nls_tolower(const struct nls_table *t,
+					unsigned char c)
 {
-	unsigned char nc = t->charset2lower[c];
+	unsigned char nc = t->ops->lowercase(t, c);
 
 	return nc ? nc : c;
 }
 
-static inline unsigned char nls_toupper(struct nls_table *t, unsigned char c)
+static inline unsigned char nls_toupper(const struct nls_table *t,
+					unsigned char c)
 {
-	unsigned char nc = t->charset2upper[c];
+	unsigned char nc = t->ops->uppercase(t, c);
 
 	return nc ? nc : c;
 }
Signature · 1 line
2.20.0.rc2
#9 lore raw

PATCH v4 9/23 nls: Add new interface for string comparisons

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

The existing stricmp() interface is limited by not accepting separated length parameters for each string being compared. This is a problem for charsets doing normalization or full casefold comparison, since different sized strings can still be matched. To resolve this problem, this patch implements a new interface, allowing charsets to do the comparison, if needed.

The original stricmp is left in the code, until we convert all caller to the new interface. Nevertheless, it was reimplemented using the new interface.

---
 include/linux/nls.h | 69 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 3 deletions(-)
include/linux/nls.h +66 -3
index c43746bd390e..980103d4c363 100644
@@ -3,6 +3,7 @@
 #define _LINUX_NLS_H
 
 #include <linux/init.h>
+#include <linux/string.h>
 
 /* Unicode has changed over the years.  Unicode code points no longer
  * fit into 16 bits; as of Unicode 5 valid code points range from 0
@@ -38,6 +39,32 @@ struct nls_ops {
 	 **/
 	int (*validate)(const struct nls_table *charset,
 			const unsigned char *str, size_t len);
+	/**
+	 * @strncmp:
+	 *
+	 * strncmp is the function for case-sensitive string comparison.
+	 * It only needs to be implemented by charsets that want to do
+	 * some fancy comparisons, like normalization-insensitive.
+	 *
+	 * Returns 0 if str1 and str2 are equal, otherwise return
+	 * non-zero.
+	 **/
+	int (*strncmp)(const struct nls_table *charset,
+		       const unsigned char *str1, size_t len1,
+		       const unsigned char *str2, size_t len2);
+
+	/**
+	 * @strncasecmp:
+	 *
+	 * strncasecmp is the function for case-insensitive string
+	 * comparison.
+	 *
+	 * Returns 0 if str1 and str2 are equal, otherwise return
+	 * non-zero.
+	 **/
+	int (*strncasecmp)(const struct nls_table *charset,
+			   const unsigned char *str1, size_t len1,
+			   const unsigned char *str2, size_t len2);
 	unsigned char (*lowercase)(const struct nls_table *charset,
 				   unsigned int c);
 	unsigned char (*uppercase)(const struct nls_table *charset,
@@ -139,10 +166,21 @@ static inline unsigned char nls_toupper(const struct nls_table *t,
 	return nc ? nc : c;
 }
 
-static inline int nls_strnicmp(struct nls_table *t, const unsigned char *s1,
-		const unsigned char *s2, int len)
+static inline int nls_strncasecmp(struct nls_table *t,
+				  const unsigned char *s1, size_t len1,
+				  const unsigned char *s2, size_t len2)
 {
-	while (len--) {
+	if (t->ops->strncasecmp)
+		return t->ops->strncasecmp(t, s1, len1, s2, len2);
+
+	if (IS_STRICT_MODE(t) &&
+	    (nls_validate(t, s1, len1) || nls_validate(t, s1, len1)))
+		return -EINVAL;
+
+	if (len1 != len2)
+		return 1;
+
+	while (len1--) {
 		if (nls_tolower(t, *s1++) != nls_tolower(t, *s2++))
 			return 1;
 	}
@@ -150,6 +188,31 @@ static inline int nls_strnicmp(struct nls_table *t, const unsigned char *s1,
 	return 0;
 }
 
+static inline int nls_strncmp(struct nls_table *t,
+			      const unsigned char *s1, size_t len1,
+			      const unsigned char *s2, size_t len2)
+{
+	if (t->ops->strncmp)
+		return t->ops->strncmp(t, s1, len1, s2, len2);
+
+	if (IS_STRICT_MODE(t) &&
+	    (nls_validate(t, s1, len1) || nls_validate(t, s1, len1)))
+		return -EINVAL;
+
+	if (len1 != len2)
+		return 1;
+
+	/* strnicmp did not return negative values. So let's keep the
+	 * abi for now */
+	return !!memcmp(s1, s2, len1);
+}
+
+static inline int nls_strnicmp(struct nls_table *t, const unsigned char *s1,
+		const unsigned char *s2, int len)
+{
+	return nls_strncasecmp(t, s1, len, s2, len);
+}
+
 /*
  * nls_nullsize - return length of null character for codepage
  * @codepage - codepage for which to return length of NULL terminator
Signature · 1 line
2.20.0.rc2
#10 lore raw

PATCH v4 10/23 nls: Add optional normalization and casefold hooks

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

The Normalization operation applies a transformation to strings to obtain the normalization form, which allow the user to determine whether any two strings are equivalent to each other. The NLS subsystem doesn't impose any constraint on what means to be equivalent, for any charsets. Unicode-based charsets, for instance, are free to support one, a few or all kinds of Unicode equivalences.

The Casefold operation is similar to Normalization, in a sense that it also allows the caller to identify equivalent strings, but it disregards case, making it ideal for case insensitive comparisons.

Default implementation are provided by the nls core, such that existing charsets can operate on the new interface. The Normalization default operation is the format NLS_NORMALIZATION_TYPE_PLAIN, which returns the identity of the string, which means no normalization. The casefold default is NLS_CASEFOLD_TYPE_TOUPPER, which returns the string with all characters converted to uppercase.

Changes since V1:
- Add default operations for casefold and normalization

---
 fs/nls/nls_core.c   |  11 +++++
 include/linux/nls.h | 116 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 127 insertions(+)
fs/nls/nls_core.c +11 -0
index 49a15bb2174f..c49088f36f4c 100644
@@ -25,6 +25,17 @@ static int nls_validate_flags(struct nls_table *table, unsigned int flags)
 	if (flags & NLS_STRICT_MODE && !table->ops->validate)
 		return -1;
 
+	if ((flags & NLS_NORMALIZATION_TYPE_MASK) && !table->ops->normalize)
+		return -1;
+
+	if ((flags & NLS_CASEFOLD_TYPE_MASK) && !table->ops->casefold)
+		return -1;
+
+	/* Reject unused flags */
+	if (flags & ~(NLS_CASEFOLD_TYPE_MASK | NLS_NORMALIZATION_TYPE_MASK |
+		      NLS_STRICT_MODE))
+		return -1;
+
 	return 0;
 }
 
include/linux/nls.h +116 -0

147 lines not on this page · raw message

Signature · 1 line
2.20.0.rc2
#11 lore raw

PATCH v4 11/23 nls: ascii: Support validation and normalization operations

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

validation is trivial. Any byte that has the MSB set is an invalid sequence.

Casefold can be implemented with uppercase or lowercase, and we have no specification on that. Callers should be safe using either of them, as long as it doesn't change.

---
 fs/nls/nls_ascii.c  | 50 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nls.h |  8 ++++++++
 2 files changed, 58 insertions(+)
fs/nls/nls_ascii.c +50 -0
index 2f4826478d3d..079a1574c19d 100644
@@ -12,6 +12,7 @@
 #include <linux/string.h>
 #include <linux/nls.h>
 #include <linux/errno.h>
+#include <linux/slab.h>
 
 static const wchar_t charset2uni[256] = {
 	/* 0x00*/
@@ -117,6 +118,8 @@ static const unsigned char charset2upper[256] = {
 	0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, /* 0x78-0x7f */
 };
 
+#define VALID_ASCII(c) (c < 128)
+
 static int uni2char(wchar_t uni, unsigned char *out, int boundlen)
 {
 	const unsigned char *uni2charset;
@@ -142,6 +145,16 @@ static int char2uni(const unsigned char *rawstring, int boundlen, wchar_t *uni)
 	return 1;
 }
 
+static int ascii_validate(const struct nls_table *table,
+			  const unsigned char *str, size_t len)
+{
+	int i;
+	for (i = 0; i < len && str[i]; i++)
+		if (!VALID_ASCII(str[i]))
+			return -1;
+	return 0;
+}
+
 static unsigned char charset_tolower(const struct nls_table *table,
 				     unsigned int c){
 	return charset2lower[c];
@@ -152,11 +165,36 @@ static unsigned char charset_toupper(const struct nls_table *table,
 	return charset2upper[c];
 }
 
+static int ascii_casefold(const struct nls_table *charset,
+			  const unsigned char *str, size_t len,
+			  unsigned char *dest, size_t dlen)
+{
+	unsigned int i;
+
+	if (dlen < len)
+		return -EINVAL;
+
+	for (i = 0; i < len; i++) {
+		if (IS_STRICT_MODE(charset) && !VALID_ASCII(str[i]))
+			return -EINVAL;
+
+		if (IS_CASEFOLD_TYPE_ASCII_TOLOWER(charset))
+			dest[i] = charset_tolower(charset, str[i]);
+		else
+			dest[i] = charset_toupper(charset, str[i]);
+	}
+	dest[len] = '\0';
+
+	return len;
+}
+
 static const struct nls_ops charset_ops = {
+	.validate = ascii_validate,
 	.lowercase = charset_toupper,
 	.uppercase = charset_tolower,
 	.uni2char = uni2char,
 	.char2uni = char2uni,
+	.casefold = ascii_casefold,
 };
 
 static struct nls_charset nls_charset;
@@ -165,9 +203,21 @@ static struct nls_table table = {
 	.ops = &charset_ops,
 };
 
+struct nls_table *ascii_load_table(const char *version, unsigned int flags)
+{
+	if (flags & ~(NLS_STRICT_MODE) ||
+	    (flags & NLS_NORMALIZATION_TYPE_MASK) != NLS_NORMALIZATION_TYPE_PLAIN)
+		return ERR_PTR(-EINVAL);
+
+	table.flags = flags;
+	return &table;
+}
+
+
 static struct nls_charset nls_charset = {
 	.charset = "ascii",
 	.tables = &table,
+	.load_table = ascii_load_table,
 };
 
 static int __init init_nls_ascii(void)
include/linux/nls.h +8 -0
index 44a06a9c69e7..aab60d4858ee 100644
@@ -178,6 +178,14 @@ IS_CASEFOLD_TYPE_##charset##_##type(const struct nls_table *c)		\
 NLS_NORMALIZATION_FUNCS(ALL, PLAIN, NLS_NORMALIZATION_TYPE_PLAIN)
 NLS_CASEFOLD_FUNCS(ALL, TOUPPER, NLS_CASEFOLD_TYPE_TOUPPER)
 
+/* ASCII */
+
+#define NLS_ASCII_CASEFOLD_TOUPPER NLS_CASEFOLD_TYPE_TOUPPER
+#define NLS_ASCII_CASEFOLD_TOLOWER NLS_CASEFOLD_TYPE(1)
+
+NLS_CASEFOLD_FUNCS(ASCII, TOUPPER, NLS_ASCII_CASEFOLD_TOUPPER)
+NLS_CASEFOLD_FUNCS(ASCII, TOLOWER, NLS_ASCII_CASEFOLD_TOLOWER)
+
 /* nls_base.c */
 extern int __register_nls(struct nls_charset *, struct module *);
 extern int unregister_nls(struct nls_charset *);
Signature · 1 line
2.20.0.rc2
#12 lore raw

PATCH v4 12/23 nls: utf8: Add unicode character database files

In reply to Gabriel Krisman Bertazi

From: Olaf Weber <[email protected]>

Add files from the Unicode Character Database, version 11.0, to the source. A helper program that generates a trie used for normalization from these files is part of a separate commit.

- Notes on the update from 8.0.0 and 11.0:

The structure of ucd files and special cases have not experienced any changes between versions 8.0.0 and 11.0.0. 8.0.0 saw the addition of Cherokee LC characters, which is an interesting case for case-folding. The update is accompanied by new tests on the test_ucd module to catch specific cases. No changes to mkutf8data script was required for the update.

The actual files are not part of the commit submitted to the list because they are to big and would bounce. Still, they can be obtained by the following script:

FILES="CaseFolding.txt DerivedAge.txt extracted/DerivedCombiningClass.txt
DerivedCoreProperties.txt NormalizationCorrections.txt NormalizationTest.txt UnicodeData.txt"
VERSION=11.0.0 BASE=http://www.unicode.org/Public/${VERSION}/ucd

for i in ${FILES} ; do
wget "${BASE}/$i" -O fs/nls/ucd/$(basename ${i} .txt)-${VERSION}.txt
done

[Move ucd directory to fs/nls/]
[Update to Unicode 11.0.0]
---
 fs/nls/ucd/README | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 fs/nls/ucd/README
fs/nls/ucd/README new file +34 -0
new file mode 100644
index 000000000000..553ce7e4c224
@@ -0,0 +1,34 @@
+The files in this directory are part of the Unicode Character Database
+for version 11.0.0 of the Unicode standard.
+
+The full set of files can be found here:
+
+  http://www.unicode.org/Public/11.0.0/ucd/
+
+The latest released version of the UCD can be found here:
+
+  http://www.unicode.org/Public/UCD/latest/
+
+The files in this directory are identical, except that they have been
+renamed with a suffix indicating the unicode version.
+
+Individual source links:
+
+  http://www.unicode.org/Public/11.0.0/ucd/CaseFolding.txt
+  http://www.unicode.org/Public/11.0.0/ucd/DerivedAge.txt
+  http://www.unicode.org/Public/11.0.0/ucd/extracted/DerivedCombiningClass.txt
+  http://www.unicode.org/Public/11.0.0/ucd/DerivedCoreProperties.txt
+  http://www.unicode.org/Public/11.0.0/ucd/NormalizationCorrections.txt
+  http://www.unicode.org/Public/11.0.0/ucd/NormalizationTest.txt
+  http://www.unicode.org/Public/11.0.0/ucd/UnicodeData.txt
+
+md5sums
+
+  414436796cf097df55f798e1585448ee  CaseFolding-11.0.0.txt
+  6032a595fbb782694456491d86eecfac  DerivedAge-11.0.0.txt
+  3240997d671297ac754ab0d27577acf7  DerivedCombiningClass-11.0.0.txt
+  d41d8cd98f00b204e9800998ecf8427e  DerivedCombiningClass.txt
+  2a4fe257d9d8184518e036194d2248ec  DerivedCoreProperties-11.0.0.txt
+  4e7d383fa0dd3cd9d49d64e5b7b7c9e0  NormalizationCorrections-11.0.0.txt
+  c9500c5b8b88e584469f056023ecc3f2  NormalizationTest-11.0.0.txt
+  acc291106c3758d2025f8d7bd5518bee  UnicodeData-11.0.0.txt
Signature · 1 line
2.20.0.rc2
#13 lore raw

PATCH v4 13/23 scripts: add trie generator for UTF-8

In reply to Gabriel Krisman Bertazi

From: Olaf Weber <[email protected]>

mkutf8data.c is the source for a program that generates utf8data.h, which contains the trie that utf8norm.c uses. The trie is generated from the Unicode 11.0.0 data files. The format of the utf8data[] table is described in utf8norm.c, which is added in the next patch.

[Rebase to mainline]
[Fix out-of-tree-build]
[Fix checkpatch warnings]
[Merge back robustness fixes from original patch. Requested by Dave Chinner]
[Update makefile to build 11.0.0 ucd files]
[drop references to xfs]
---
 fs/nls/Kconfig       |   10 +
 fs/nls/Makefile      |   13 +
 scripts/Makefile     |    1 +
 scripts/mkutf8data.c | 3168 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 3192 insertions(+)
 create mode 100644 scripts/mkutf8data.c
fs/nls/Kconfig +10 -0
index e2ce79ef48c4..7cb2848da608 100644
@@ -616,4 +616,14 @@ config NLS_UTF8
 	  input/output character sets. Say Y here for the UTF-8 encoding of
 	  the Unicode/ISO9646 universal character set.
 
+#
+# utf8 normalization
+#
+config NLS_UTF8_NORMALIZATION
+	bool "UTF-8 normalization and casefolding support"
+	depends on NLS_UTF8
+	help
+	  Say Y here to enable utf8 NFKD normalization and casefolding
+	  support.
+
 endif # NLS
fs/nls/Makefile +13 -0
index 5f42ceff9d15..840e06aefd47 100644
@@ -55,3 +55,16 @@ obj-$(CONFIG_NLS_MAC_INUIT)     += mac-inuit.o
 obj-$(CONFIG_NLS_MAC_ROMANIAN)  += mac-romanian.o
 obj-$(CONFIG_NLS_MAC_ROMAN)     += mac-roman.o
 obj-$(CONFIG_NLS_MAC_TURKISH)   += mac-turkish.o
+
+$(obj)/utf8data.h: $(srctree)/$(src)/ucd/*.txt $(objtree)/scripts/mkutf8data FORCE
+	$(call cmd,mkutf8data)
+quiet_cmd_mkutf8data = MKUTF8DATA $@
+      cmd_mkutf8data = $(objtree)/scripts/mkutf8data \
+		-a $(srctree)/$(src)/ucd/DerivedAge-11.0.0.txt \
+		-c $(srctree)/$(src)/ucd/DerivedCombiningClass-11.0.0.txt \
+		-p $(srctree)/$(src)/ucd/DerivedCoreProperties-11.0.0.txt \
+		-d $(srctree)/$(src)/ucd/UnicodeData-11.0.0.txt \
+		-f $(srctree)/$(src)/ucd/CaseFolding-11.0.0.txt \
+		-n $(srctree)/$(src)/ucd/NormalizationCorrections-11.0.0.txt \
+		-t $(srctree)/$(src)/ucd/NormalizationTest-11.0.0.txt \
+		-o $@
scripts/Makefile +1 -0
index ece52ff20171..b36208c62c17 100644
@@ -20,6 +20,7 @@ hostprogs-$(CONFIG_ASN1)	 += asn1_compiler
 hostprogs-$(CONFIG_MODULE_SIG)	 += sign-file
 hostprogs-$(CONFIG_SYSTEM_TRUSTED_KEYRING) += extract-cert
 hostprogs-$(CONFIG_SYSTEM_EXTRA_CERTIFICATE) += insert-sys-cert
+hostprogs-$(CONFIG_NLS_UTF8_NORMALIZATION) += mkutf8data
 
 HOSTCFLAGS_sortextable.o = -I$(srctree)/tools/include
 HOSTCFLAGS_asn1_compiler.o = -I$(srctree)/include
scripts/mkutf8data.c new file +3168 -0

2,963 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.0.rc2

Part of the diff is not shown here. View the raw message.

#14 lore raw

PATCH v4 14/23 nls: utf8: Move nls-utf8{,-core}.c

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

nls_utf8 will be generated from multiple files, so lets move the existing code to a -core suffix.

---
 fs/nls/Makefile                        | 3 +++
 fs/nls/{nls_utf8.c => nls_utf8-core.c} | 0
 2 files changed, 3 insertions(+)
 rename fs/nls/{nls_utf8.c => nls_utf8-core.c} (100%)
fs/nls/Makefile +3 -0
index 840e06aefd47..c94221b6108d 100644
@@ -43,7 +43,10 @@ obj-$(CONFIG_NLS_ISO8859_14)	+= nls_iso8859-14.o
 obj-$(CONFIG_NLS_ISO8859_15)	+= nls_iso8859-15.o
 obj-$(CONFIG_NLS_KOI8_R)	+= nls_koi8-r.o
 obj-$(CONFIG_NLS_KOI8_U)	+= nls_koi8-u.o nls_koi8-ru.o
+
 obj-$(CONFIG_NLS_UTF8)		+= nls_utf8.o
+nls_utf8-y			+= nls_utf8-core.o
+
 obj-$(CONFIG_NLS_MAC_CELTIC)    += mac-celtic.o
 obj-$(CONFIG_NLS_MAC_CENTEURO)  += mac-centeuro.o
 obj-$(CONFIG_NLS_MAC_CROATIAN)  += mac-croatian.o
fs/nls/nls_utf8-core.c renamed from fs/nls/nls_utf8.c +0 -0
similarity index 100%
rename from fs/nls/nls_utf8.c
rename to fs/nls/nls_utf8-core.c
Signature · 1 line
2.20.0.rc2
#15 lore raw

PATCH v4 15/23 nls: utf8: Introduce code for UTF-8 normalization

In reply to Gabriel Krisman Bertazi

From: Olaf Weber <[email protected]>

Supporting functions for UTF-8 normalization are in utf8norm.c with the header utf8norm.h. Two normalization forms are supported: nfkdi and nfkdicf.

nfkdi:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.

nfkdicf:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
- Apply a full casefold (C + F).

For the purposes of the code, a string is valid UTF-8 if:

- The values encoded are 0x1..0x10FFFF.
- The surrogate codepoints 0xD800..0xDFFFF are not encoded.
- The shortest possible encoding is used for all values.

The supporting functions work on null-terminated strings (utf8 prefix) and on length-limited strings (utf8n prefix).

Quoted 1 line

From the original SGI patch and for conformity with coding standards,

the utf8data_t typedef was dropped, since it was just masking the struct keyword. On other occasions, namely utf8leaf_t and utf8trie_t, I decided to keep it, since they are simple pointers to memory buffers, and using uchars here wouldn't provide any more meaningful information.

Changes since RFC v2:
- Merge to NLS system

Changes since RFC v1:
- utf8_version_is_supported receives maj, min and rev as separate arguments. (Olaf Weber)

[Rebase to Mainline]
[Fix up checkpatch.pl warnings]
[Drop typedefs]
[Merge with NLS subsystem]
---
 fs/nls/Makefile        |   2 +
 fs/nls/nls_utf8-norm.c | 640 +++++++++++++++++++++++++++++++++++++++++
 fs/nls/utf8n.h         | 112 ++++++++
 3 files changed, 754 insertions(+)
 create mode 100644 fs/nls/nls_utf8-norm.c
 create mode 100644 fs/nls/utf8n.h
fs/nls/Makefile +2 -0
index c94221b6108d..bd13c1a90767 100644
@@ -46,6 +46,7 @@ obj-$(CONFIG_NLS_KOI8_U)	+= nls_koi8-u.o nls_koi8-ru.o
 
 obj-$(CONFIG_NLS_UTF8)		+= nls_utf8.o
 nls_utf8-y			+= nls_utf8-core.o
+nls_utf8-$(CONFIG_NLS_UTF8_NORMALIZATION) += nls_utf8-norm.o
 
 obj-$(CONFIG_NLS_MAC_CELTIC)    += mac-celtic.o
 obj-$(CONFIG_NLS_MAC_CENTEURO)  += mac-centeuro.o
@@ -59,6 +60,7 @@ obj-$(CONFIG_NLS_MAC_ROMANIAN)  += mac-romanian.o
 obj-$(CONFIG_NLS_MAC_ROMAN)     += mac-roman.o
 obj-$(CONFIG_NLS_MAC_TURKISH)   += mac-turkish.o
 
+$(obj)/nls_utf8-norm.o: $(obj)/utf8data.h
 $(obj)/utf8data.h: $(srctree)/$(src)/ucd/*.txt $(objtree)/scripts/mkutf8data FORCE
 	$(call cmd,mkutf8data)
 quiet_cmd_mkutf8data = MKUTF8DATA $@
fs/nls/nls_utf8-norm.c new file +640 -0

640 lines not on this page · raw message

fs/nls/utf8n.h new file +112 -0

112 lines not on this page · raw message

Signature · 1 line
2.20.0.rc2
#16 lore raw

PATCH v4 16/23 nls: utf8n: reduce the size of utf8data[]

In reply to Gabriel Krisman Bertazi

From: Olaf Weber <[email protected]>

Remove the Hangul decompositions from the utf8data trie, and do algorithmic decomposition to calculate them on the fly. To store the decomposition the caller of utf8lookup()/utf8nlookup() must provide a 12-byte buffer, which is used to synthesize a leaf with the decomposition. Trie size is reduced from 245kB to 90kB.

[Rebase to mainline]
[Fix checkpatch errors]

[Extract robustness fixes and merge back to original mkutf8data.c patch]

---
 fs/nls/nls_utf8-norm.c | 191 +++++++++++++++++++++++---
 fs/nls/utf8n.h         |   4 +
 scripts/mkutf8data.c   | 298 ++++++++++++++++++++++++++++++++++++-----
 3 files changed, 436 insertions(+), 57 deletions(-)
fs/nls/nls_utf8-norm.c +171 -20
index ca0bbf644b49..64c3cc74a2ca 100644
@@ -98,6 +98,38 @@ static inline int utf8clen(const char *s)
 	return 1 + (c >= 0xC0) + (c >= 0xE0) + (c >= 0xF0);
 }
 
+/*
+ * Decode a 3-byte UTF-8 sequence.
+ */
+static unsigned int
+utf8decode3(const char *str)
+{
+	unsigned int		uc;
+
+	uc = *str++ & 0x0F;
+	uc <<= 6;
+	uc |= *str++ & 0x3F;
+	uc <<= 6;
+	uc |= *str++ & 0x3F;
+
+	return uc;
+}
+
+/*
+ * Encode a 3-byte UTF-8 sequence.
+ */
+static int
+utf8encode3(char *str, unsigned int val)
+{
+	str[2] = (val & 0x3F) | 0x80;
+	val >>= 6;
+	str[1] = (val & 0x3F) | 0x80;
+	val >>= 6;
+	str[0] = val | 0xE0;
+
+	return 3;
+}
+
 /*
  * utf8trie_t
  *
@@ -159,7 +191,8 @@ typedef const unsigned char utf8trie_t;
  *          characters with the Default_Ignorable_Code_Point property.
  *          These do affect normalization, as they all have CCC 0.
  *
- * The decompositions in the trie have been fully expanded.
+ * The decompositions in the trie have been fully expanded, with the
+ * exception of Hangul syllables, which are decomposed algorithmically.
  *
  * Casefolding, if applicable, is also done using decompositions.
  *
@@ -179,6 +212,105 @@ typedef const unsigned char utf8leaf_t;
 #define STOPPER		(0)
 #define	DECOMPOSE	(255)
 
+/* Marker for hangul syllable decomposition. */
+#define HANGUL		((char)(255))
+/* Size of the synthesized leaf used for Hangul syllable decomposition. */
+#define UTF8HANGULLEAF	(12)
+
+/*
+ * Hangul decomposition (algorithm from Section 3.12 of Unicode 6.3.0)
+ *
+ * AC00;<Hangul Syllable, First>;Lo;0;L;;;;;N;;;;;
+ * D7A3;<Hangul Syllable, Last>;Lo;0;L;;;;;N;;;;;
+ *
+ * SBase = 0xAC00
+ * LBase = 0x1100
+ * VBase = 0x1161
+ * TBase = 0x11A7
+ * LCount = 19
+ * VCount = 21
+ * TCount = 28
+ * NCount = 588 (VCount * TCount)
+ * SCount = 11172 (LCount * NCount)
+ *
+ * Decomposition:
+ *   SIndex = s - SBase
+ *
+ * LV (Canonical/Full)
+ *   LIndex = SIndex / NCount
+ *   VIndex = (Sindex % NCount) / TCount
+ *   LPart = LBase + LIndex
+ *   VPart = VBase + VIndex
+ *
+ * LVT (Canonical)
+ *   LVIndex = (SIndex / TCount) * TCount
+ *   TIndex = (Sindex % TCount)
+ *   LVPart = SBase + LVIndex
+ *   TPart = TBase + TIndex
+ *
+ * LVT (Full)
+ *   LIndex = SIndex / NCount
+ *   VIndex = (Sindex % NCount) / TCount
+ *   TIndex = (Sindex % TCount)
+ *   LPart = LBase + LIndex
+ *   VPart = VBase + VIndex
+ *   if (TIndex == 0) {
+ *          d = <LPart, VPart>
+ *   } else {
+ *          TPart = TBase + TIndex
+ *          d = <LPart, TPart, VPart>
+ *   }
+ */
+
+/* Constants */
+#define SB	(0xAC00)
+#define LB	(0x1100)
+#define VB	(0x1161)
+#define TB	(0x11A7)
+#define LC	(19)
+#define VC	(21)
+#define TC	(28)
+#define NC	(VC * TC)
+#define SC	(LC * NC)
+
+/* Algorithmic decomposition of hangul syllable. */
+static utf8leaf_t *
+utf8hangul(const char *str, unsigned char *hangul)
+{
+	unsigned int	si;
+	unsigned int	li;
+	unsigned int	vi;
+	unsigned int	ti;
+	unsigned char	*h;
+
+	/* Calculate the SI, LI, VI, and TI values. */
+	si = utf8decode3(str) - SB;
+	li = si / NC;
+	vi = (si % NC) / TC;
+	ti = si % TC;
+
+	/* Fill in base of leaf. */
+	h = hangul;
+	LEAF_GEN(h) = 2;
+	LEAF_CCC(h) = DECOMPOSE;
+	h += 2;
+
+	/* Add LPart, a 3-byte UTF-8 sequence. */
+	h += utf8encode3((char *)h, li + LB);
+
+	/* Add VPart, a 3-byte UTF-8 sequence. */
+	h += utf8encode3((char *)h, vi + VB);
+
+	/* Add TPart if required, also a 3-byte UTF-8 sequence. */
+	if (ti)
+		h += utf8encode3((char *)h, ti + TB);
+
+	/* Terminate string. */
+	h[0] = '\0';
+
+	return hangul;
+}
+
 /*
  * Use trie to scan s, touching at most len bytes.
  * Returns the leaf if one exists, NULL otherwise.
@@ -187,8 +319,8 @@ typedef const unsigned char utf8leaf_t;
  * is well-formed and corresponds to a known unicode code point.  The
  * shorthand for this will be "is valid UTF-8 unicode".
  */
-static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s,
-			       size_t len)
+static utf8leaf_t *utf8nlookup(const struct utf8data *data,
+			       unsigned char *hangul, const char *s, size_t len)
 {
 	utf8trie_t	*trie = utf8data + data->offset;
 	int		offlen;
@@ -226,8 +358,7 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s,
 				trie++;
 			} else {
 				/* No right node. */
-				node = 0;
-				trie = NULL;
+				return NULL;
 			}
 		} else {
 			/* Left leg */
@@ -237,8 +368,7 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s,
 				trie += offlen + 1;
 			} else if (*trie & RIGHTPATH) {
 				/* No left node. */
-				node = 0;
-				trie = NULL;
+				return NULL;
 			} else {
 				/* Left node after this node */
 				node = (*trie & TRIENODE);
@@ -246,6 +376,14 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s,
 			}
 		}
 	}
+	/*
+	 * Hangul decomposition is done algorithmically. These are the
+	 * codepoints >= 0xAC00 and <= 0xD7A3. Their UTF-8 encoding is
+	 * always 3 bytes long, so s has been advanced twice, and the
+	 * start of the sequence is at s-2.
+	 */
+	if (LEAF_CCC(trie) == DECOMPOSE && LEAF_STR(trie)[0] == HANGUL)
+		trie = utf8hangul(s - 2, hangul);
 	return trie;
 }
 
@@ -255,9 +393,10 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data, const char *s,
  *
  * Forwards to utf8nlookup().
  */
-static utf8leaf_t *utf8lookup(const struct utf8data *data, const char *s)
+static utf8leaf_t *utf8lookup(const struct utf8data *data,
+			      unsigned char *hangul, const char *s)
 {
-	return utf8nlookup(data, s, (size_t)-1);
+	return utf8nlookup(data, hangul, s, (size_t)-1);
 }
 
 /*
@@ -270,11 +409,13 @@ int utf8agemax(const struct utf8data *data, const char *s)
 	utf8leaf_t	*leaf;
 	int		age = 0;
 	int		leaf_age;
+	unsigned char	hangul[UTF8HANGULLEAF];
 
 	if (!data)
 		return -1;
+
 	while (*s) {
-		leaf = utf8lookup(data, s);
+		leaf = utf8lookup(data, hangul, s);
 		if (!leaf)
 			return -1;
 
@@ -297,12 +438,13 @@ int utf8agemin(const struct utf8data *data, const char *s)
 	utf8leaf_t	*leaf;
 	int		age;
 	int		leaf_age;
+	unsigned char	hangul[UTF8HANGULLEAF];
 
 	if (!data)
 		return -1;
 	age = data->maxage;
 	while (*s) {
-		leaf = utf8lookup(data, s);
+		leaf = utf8lookup(data, hangul, s);
 		if (!leaf)
 			return -1;
 		leaf_age = utf8agetab[LEAF_GEN(leaf)];
@@ -323,11 +465,13 @@ int utf8nagemax(const struct utf8data *data, const char *s, size_t len)
 	utf8leaf_t	*leaf;
 	int		age = 0;
 	int		leaf_age;
+	unsigned char	hangul[UTF8HANGULLEAF];
 
 	if (!data)
 		return -1;
+
 	while (len && *s) {
-		leaf = utf8nlookup(data, s, len);
+		leaf = utf8nlookup(data, hangul, s, len);
 		if (!leaf)
 			return -1;
 		leaf_age = utf8agetab[LEAF_GEN(leaf)];
@@ -349,12 +493,13 @@ int utf8nagemin(const struct utf8data *data, const char *s, size_t len)
 	utf8leaf_t	*leaf;
 	int		leaf_age;
 	int		age;
+	unsigned char	hangul[UTF8HANGULLEAF];
 
 	if (!data)
 		return -1;
 	age = data->maxage;
 	while (len && *s) {
-		leaf = utf8nlookup(data, s, len);
+		leaf = utf8nlookup(data, hangul, s, len);
 		if (!leaf)
 			return -1;
 		leaf_age = utf8agetab[LEAF_GEN(leaf)];
@@ -377,11 +522,12 @@ ssize_t utf8len(const struct utf8data *data, const char *s)
 {
 	utf8leaf_t	*leaf;
 	size_t		ret = 0;
+	unsigned char	hangul[UTF8HANGULLEAF];
 
 	if (!data)
 		return -1;
 	while (*s) {
-		leaf = utf8lookup(data, s);
+		leaf = utf8lookup(data, hangul, s);
 		if (!leaf)
 			return -1;
 		if (utf8agetab[LEAF_GEN(leaf)] > data->maxage)
@@ -404,11 +550,12 @@ ssize_t utf8nlen(const struct utf8data *data, const char *s, size_t len)
 {
 	utf8leaf_t	*leaf;
 	size_t		ret = 0;
+	unsigned char	hangul[UTF8HANGULLEAF];
 
 	if (!data)
 		return -1;
 	while (len && *s) {
-		leaf = utf8nlookup(data, s, len);
+		leaf = utf8nlookup(data, hangul, s, len);
 		if (!leaf)
 			return -1;
 		if (utf8agetab[LEAF_GEN(leaf)] > data->maxage)
@@ -531,10 +678,12 @@ int utf8byte(struct utf8cursor *u8c)
 		}
 
 		/* Look up the data for the current character. */
-		if (u8c->p)
-			leaf = utf8lookup(u8c->data, u8c->s);
-		else
-			leaf = utf8nlookup(u8c->data, u8c->s, u8c->len);
+		if (u8c->p) {
+			leaf = utf8lookup(u8c->data, u8c->hangul, u8c->s);
+		} else {
+			leaf = utf8nlookup(u8c->data, u8c->hangul,
+					   u8c->s, u8c->len);
+		}
 
 		/* No leaf found implies that the input is a binary blob. */
 		if (!leaf)
@@ -555,7 +704,9 @@ int utf8byte(struct utf8cursor *u8c)
 				ccc = STOPPER;
 				goto ccc_mismatch;
 			}
-			leaf = utf8lookup(u8c->data, u8c->s);
+
+			leaf = utf8lookup(u8c->data, u8c->hangul, u8c->s);
+			ccc = LEAF_CCC(leaf);
 		}
 
 		/*
fs/nls/utf8n.h +4 -0
index 0f5fc14d4fd2..f60827663503 100644
@@ -76,6 +76,9 @@ extern int utf8nagemin(const struct utf8data *data, const char *s, size_t len);
 extern ssize_t utf8len(const struct utf8data *data, const char *s);
 extern ssize_t utf8nlen(const struct utf8data *data, const char *s, size_t len);
 
+/* Needed in struct utf8cursor below. */
+#define UTF8HANGULLEAF	(12)
+
 /*
  * Cursor structure used by the normalizer.
  */
@@ -89,6 +92,7 @@ struct utf8cursor {
 	unsigned int	slen;
 	short int	ccc;
 	short int	nccc;
+	unsigned char	hangul[UTF8HANGULLEAF];
 };
 
 /*
scripts/mkutf8data.c +261 -37

601 lines not on this page · raw message

Signature · 1 line
2.20.0.rc2
#17 lore raw

PATCH v4 17/23 nls: utf8: Integrate utf8 normalization code with utf8 charset

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

This patch integrates the utf8n patches with the NLS utf8 charset by implementing the nls_ops operations and nls_charset table. The Normalization is done with NFKD, and Casefold is implemented using the NFKD+CF algorithm, implemented by Olaf Weber and SGI. The high level, strcmp, strncmp functions are implemented on top of the same utf8 code.

Utf-8 with normalization is exposed as optional on top of the existing utf8 charset, and disabled by default, to avoid changing the behavior of existing nls_utf8 users. To enable normalization, the specific normalization type must be set at load_table() time.

Changes since RFC v2:
- Integrate with NLS
- Merge utf8n with nls_utf8.

Changes since RFC v1:
- Change error return code from EIO to EINVAL. (Olaf Weber)
- Fix issues with strncmp/strcmp. (Olaf Weber)
- Remove stack buffer in normalization/casefold. (Olaf Weber)
- Include length parameter for second string on comparison functions.
- Change length type to size_t.

---
 fs/nls/nls_utf8-core.c | 269 ++++++++++++++++++++++++++++++++++++++---
 fs/nls/nls_utf8-norm.c |   6 +
 fs/nls/utf8n.h         |   1 +
 include/linux/nls.h    |   8 ++
 4 files changed, 270 insertions(+), 14 deletions(-)
fs/nls/nls_utf8-core.c +255 -14
index fe1ac5efaa37..1b7320bd9c34 100644
@@ -6,10 +6,15 @@
 #include <linux/module.h>
 #include <linux/kernel.h>
 #include <linux/string.h>
+#include <linux/slab.h>
+#include <linux/parser.h>
 #include <linux/nls.h>
 #include <linux/errno.h>
 
+#include "utf8n.h"
+
 static unsigned char identity[256];
+static struct nls_charset utf8_info;
 
 static int uni2char(wchar_t uni, unsigned char *out, int boundlen)
 {
@@ -50,22 +55,257 @@ static unsigned char charset_toupper(const struct nls_table *table,
 	return identity[c];
 }
 
-static const struct nls_ops charset_ops = {
-	.lowercase = charset_toupper,
-	.uppercase = charset_tolower,
-	.uni2char = uni2char,
-	.char2uni = char2uni,
-};
+#ifdef CONFIG_NLS_UTF8_NORMALIZATION
+
+static int utf8_validate(const struct nls_table *charset,
+			 const unsigned char *str, size_t len)
+{
+	const struct utf8data *data = utf8nfkdi(charset->version);
+
+	if (utf8nlen(data, str, len) < 0)
+		return -1;
+	return 0;
+}
+
+static int utf8_strncmp(const struct nls_table *charset,
+			const unsigned char *str1, size_t len1,
+			const unsigned char *str2, size_t len2)
+{
+	const struct utf8data *data = utf8nfkdi(charset->version);
+	struct utf8cursor cur1, cur2;
+	int c1, c2;
+
+	if (utf8ncursor(&cur1, data, str1, len1) < 0)
+		goto invalid_seq;
+
+	if (utf8ncursor(&cur2, data, str2, len2) < 0)
+		goto invalid_seq;
+
+	do {
+		c1 = utf8byte(&cur1);
+		c2 = utf8byte(&cur2);
+
+		if (c1 < 0 || c2 < 0)
+			goto invalid_seq;
+		if (c1 != c2)
+			return 1;
+	} while (c1);
+
+	return 0;
+
+invalid_seq:
+	if(IS_STRICT_MODE(charset))
+		return -EINVAL;
+
+	/* Treat the sequence as a binary blob. */
+	if (len1 != len2)
+		return 1;
+
+	return !!memcmp(str1, str2, len1);
+}
+
+static int utf8_strncasecmp(const struct nls_table *charset,
+			    const unsigned char *str1, size_t len1,
+			    const unsigned char *str2, size_t len2)
+{
+	const struct utf8data *data = utf8nfkdicf(charset->version);
+	struct utf8cursor cur1, cur2;
+	int c1, c2;
+
+	if (utf8ncursor(&cur1, data, str1, len1) < 0)
+		goto invalid_seq;
+
+	if (utf8ncursor(&cur2, data, str2, len2) < 0)
+		goto invalid_seq;
+
+	do {
+		c1 = utf8byte(&cur1);
+		c2 = utf8byte(&cur2);
+
+		if (c1 < 0 || c2 < 0)
+			goto invalid_seq;
+		if (c1 != c2)
+			return 1;
+	} while (c1);
+
+	return 0;
+
+invalid_seq:
+	if(IS_STRICT_MODE(charset))
+		return -EINVAL;
+
+	/* Treat the sequence as a binary blob. */
+	if (len1 != len2)
+		return 1;
+
+	return !!memcmp(str1, str2, len1);
+}
+
+static int utf8_casefold_nfkdcf(const struct nls_table *charset,
+				const unsigned char *str, size_t len,
+				unsigned char *dest, size_t dlen)
+{
+	const struct utf8data *data = utf8nfkdicf(charset->version);
+	struct utf8cursor cur;
+	size_t nlen = 0;
+
+	if (utf8ncursor(&cur, data, str, len) < 0)
+		goto invalid_seq;
+
+	for (nlen = 0; nlen < dlen; nlen++) {
+		dest[nlen] = utf8byte(&cur);
+		if (!dest[nlen])
+			return nlen;
+		if (dest[nlen] == -1)
+			break;
+	}
+
+invalid_seq:
+	if (IS_STRICT_MODE(charset))
+		return -EINVAL;
+
+	/* Treat the sequence as a binary blob. */
+	memcpy(dest, str, len);
+	return len;
+}
+
+static int utf8_normalize_nfkd(const struct nls_table *charset,
+			       const unsigned char *str,
+			       size_t len, unsigned char *dest, size_t dlen)
+{
+	const struct utf8data *data = utf8nfkdi(charset->version);
+	struct utf8cursor cur;
+	ssize_t nlen = 0;
+
+	if (utf8ncursor(&cur, data, str, len) < 0)
+		goto invalid_seq;
 
-static struct nls_charset nls_charset;
-static struct nls_table table = {
-	.charset = &nls_charset,
-	.ops = &charset_ops,
+	for (nlen = 0; nlen < dlen; nlen++) {
+		dest[nlen] = utf8byte(&cur);
+		if (!dest[nlen])
+			return nlen;
+		if (dest[nlen] == -1)
+			break;
+	}
+
+invalid_seq:
+	if (IS_STRICT_MODE(charset))
+		return -EINVAL;
+
+	/* Treat the sequence as a binary blob. */
+	memcpy(dest, str, len);
+	return len;
+}
+
+static int utf8_parse_version(const char *version, unsigned int *maj,
+			      unsigned int *min, unsigned int *rev)
+{
+	substring_t args[3];
+	char version_string[12];
+	const struct match_token token[] = {
+		{1, "%d.%d.%d"},
+		{0, NULL}
+	};
+
+	strncpy(version_string, version, sizeof(version_string));
+
+	if (match_token(version_string, token, args) != 1)
+		return -EINVAL;
+
+	if (match_int(&args[0], maj) || match_int(&args[1], min) ||
+	    match_int(&args[2], rev))
+		return -EINVAL;
+
+	return 0;
+}
+#endif
+
+struct utf8_table {
+	struct nls_table tbl;
+	struct nls_ops ops;
 };
 
-static struct nls_charset nls_charset = {
+static void utf8_set_ops(struct utf8_table *utbl)
+{
+	utbl->ops.lowercase = charset_toupper;
+	utbl->ops.uppercase = charset_tolower;
+	utbl->ops.uni2char = uni2char;
+	utbl->ops.char2uni = char2uni;
+
+#ifdef CONFIG_NLS_UTF8_NORMALIZATION
+	utbl->ops.validate = utf8_validate;
+
+	if (IS_NORMALIZATION_TYPE_UTF8_NFKD(&utbl->tbl)) {
+		utbl->ops.normalize = utf8_normalize_nfkd;
+		utbl->ops.strncmp = utf8_strncmp;
+	}
+
+	if (IS_CASEFOLD_TYPE_UTF8_NFKDCF(&utbl->tbl)) {
+		utbl->ops.casefold = utf8_casefold_nfkdcf;
+		utbl->ops.strncasecmp = utf8_strncasecmp;
+	}
+#endif
+
+	utbl->tbl.ops = &utbl->ops;
+}
+
+static struct nls_table *utf8_load_table(const char *version, unsigned int flags)
+{
+	struct utf8_table *utbl = NULL;
+	unsigned int nls_version;
+
+#ifdef CONFIG_NLS_UTF8_NORMALIZATION
+	if (version) {
+		unsigned int maj, min, rev;
+
+		if (utf8_parse_version(version, &maj, &min, &rev) < 0)
+			return ERR_PTR(-EINVAL);
+
+		if (!utf8version_is_supported(maj, min, rev))
+			return ERR_PTR(-EINVAL);
+
+		nls_version = UNICODE_AGE(maj, min, rev);
+	} else {
+		nls_version = utf8version_latest();
+		printk(KERN_WARNING"UTF-8 version not specified. "
+		       "Assuming latest supported version (%d.%d.%d).",
+		       (nls_version >> 16) & 0xff, (nls_version >> 8) & 0xff,
+		       (nls_version & 0xff));
+	}
+#else
+	nls_version = 0;
+#endif
+
+	utbl = kzalloc(sizeof(struct utf8_table), GFP_KERNEL);
+	if (!utbl)
+		return ERR_PTR(-ENOMEM);
+
+	utbl->tbl.charset = &utf8_info;
+	utbl->tbl.version = nls_version;
+	utbl->tbl.flags = flags;
+	utf8_set_ops(utbl);
+
+	utbl->tbl.next = utf8_info.tables;
+	utf8_info.tables = &utbl->tbl;
+
+	return &utbl->tbl;
+}
+
+static void utf8_cleanup_tables(void)
+{
+	struct nls_table *tmp, *tbl = utf8_info.tables;
+
+	while (tbl) {
+		tmp = tbl;
+		tbl = tbl->next;
+		kfree(tmp);
+	}
+	utf8_info.tables = NULL;
+}
+
+static struct nls_charset utf8_info = {
 	.charset = "utf8",
-	.tables = &table,
+	.load_table = utf8_load_table,
 };
 
 static int __init init_nls_utf8(void)
@@ -74,12 +314,13 @@ static int __init init_nls_utf8(void)
 	for (i=0; i<256; i++)
 		identity[i] = i;
 
-        return register_nls(&nls_charset);
+        return register_nls(&utf8_info);
 }
 
 static void __exit exit_nls_utf8(void)
 {
-        unregister_nls(&nls_charset);
+	unregister_nls(&utf8_info);
+	utf8_cleanup_tables();
 }
 
 module_init(init_nls_utf8)
fs/nls/nls_utf8-norm.c +6 -0
index 64c3cc74a2ca..abee8b376a87 100644
@@ -38,6 +38,12 @@ int utf8version_is_supported(u8 maj, u8 min, u8 rev)
 }
 EXPORT_SYMBOL(utf8version_is_supported);
 
+int utf8version_latest()
+{
+	return utf8vers;
+}
+EXPORT_SYMBOL(utf8version_latest);
+
 /*
  * UTF-8 valid ranges.
  *
fs/nls/utf8n.h +1 -0
index f60827663503..b4697f9bfbab 100644
@@ -32,6 +32,7 @@
 
 /* Highest unicode version supported by the data tables. */
 extern int utf8version_is_supported(u8 maj, u8 min, u8 rev);
+extern int utf8version_latest(void);
 
 /*
  * Look for the correct const struct utf8data for a unicode version.
include/linux/nls.h +8 -0
index aab60d4858ee..aee5cbfc07c6 100644
@@ -186,6 +186,14 @@ NLS_CASEFOLD_FUNCS(ALL, TOUPPER, NLS_CASEFOLD_TYPE_TOUPPER)
 NLS_CASEFOLD_FUNCS(ASCII, TOUPPER, NLS_ASCII_CASEFOLD_TOUPPER)
 NLS_CASEFOLD_FUNCS(ASCII, TOLOWER, NLS_ASCII_CASEFOLD_TOLOWER)
 
+/* UTF-8 */
+
+#define NLS_UTF8_NORMALIZATION_TYPE_NFKD NLS_NORMALIZATION_TYPE(1)
+#define NLS_UTF8_CASEFOLD_TYPE_NFKDCF	 NLS_CASEFOLD_TYPE(1)
+
+NLS_NORMALIZATION_FUNCS(UTF8, NFKD, NLS_UTF8_NORMALIZATION_TYPE_NFKD)
+NLS_CASEFOLD_FUNCS(UTF8, NFKDCF, NLS_UTF8_CASEFOLD_TYPE_NFKDCF)
+
 /* nls_base.c */
 extern int __register_nls(struct nls_charset *, struct module *);
 extern int unregister_nls(struct nls_charset *);
Signature · 1 line
2.20.0.rc2
#18 lore raw

PATCH v4 18/23 nls: utf8: Introduce test module for normalized utf8 implementation

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

This implements a in-kernel sanity test module for the utf8 normalization core. At probe time, it will run basic sequences through the utf8n core, to identify problems will equivalent sequences and normalization/casefold code. This is supposed to be useful for regression testing when adding support for a new version of utf8 to linux.

Changes since RFC v2:
- Merge with NLS

Changes since RFC v1:
- Include comparison tests for matching strings with different lengths.
- Include tests for characters included in unicode 8.0.0, 9.0.0 and 10.0.0.

---
 fs/nls/Kconfig             |   5 +
 fs/nls/Makefile            |   1 +
 fs/nls/nls_utf8-selftest.c | 316 +++++++++++++++++++++++++++++++++++++
 3 files changed, 322 insertions(+)
 create mode 100644 fs/nls/nls_utf8-selftest.c
fs/nls/Kconfig +5 -0
index 7cb2848da608..b9c9b663d0ab 100644
@@ -626,4 +626,9 @@ config NLS_UTF8_NORMALIZATION
 	  Say Y here to enable utf8 NFKD normalization and casefolding
 	  support.
 
+config NLS_UTF8_NORMALIZATION_SELFTEST
+	tristate "Test UTF-8 normalization support"
+	depends on NLS_UTF8
+	default n
+
 endif # NLS
fs/nls/Makefile +1 -0
index bd13c1a90767..e88006d2f3ac 100644
@@ -47,6 +47,7 @@ obj-$(CONFIG_NLS_KOI8_U)	+= nls_koi8-u.o nls_koi8-ru.o
 obj-$(CONFIG_NLS_UTF8)		+= nls_utf8.o
 nls_utf8-y			+= nls_utf8-core.o
 nls_utf8-$(CONFIG_NLS_UTF8_NORMALIZATION) += nls_utf8-norm.o
+obj-$(CONFIG_NLS_UTF8_NORMALIZATION_SELFTEST) += nls_utf8-selftest.o
 
 obj-$(CONFIG_NLS_MAC_CELTIC)    += mac-celtic.o
 obj-$(CONFIG_NLS_MAC_CENTEURO)  += mac-centeuro.o
fs/nls/nls_utf8-selftest.c new file +316 -0

316 lines not on this page · raw message

Signature · 1 line
2.20.0.rc2
#19 lore raw

PATCH v4 19/23 ext4: Reserve superblock fields for encoding information

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

The s_encoding field stores a magic number indicating the encoding format and version used globally by file and directory names in the filesystem.

The s_encoding_flags defines policies for using the charset encoding, like how to handle invalid sequences and what kind of normalization to use.

---
 fs/ext4/ext4.h | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
fs/ext4/ext4.h +6 -1
index 3f89d0ab08fc..52c9e8b948a0 100644
@@ -1311,7 +1311,9 @@ struct ext4_super_block {
 	__u8	s_first_error_time_hi;
 	__u8	s_last_error_time_hi;
 	__u8	s_pad[2];
-	__le32	s_reserved[96];		/* Padding to the end of the block */
+	__le16  s_encoding;		/* Filename charset encoding */
+	__le16  s_encoding_flags;	/* Filename charset encoding flags */
+	__le32	s_reserved[95];		/* Padding to the end of the block */
 	__le32	s_checksum;		/* crc32c(superblock) */
 };
 
@@ -1661,6 +1663,7 @@ static inline void ext4_clear_state_flags(struct ext4_inode_info *ei)
 #define EXT4_FEATURE_INCOMPAT_LARGEDIR		0x4000 /* >2GB or 3-lvl htree */
 #define EXT4_FEATURE_INCOMPAT_INLINE_DATA	0x8000 /* data in inode */
 #define EXT4_FEATURE_INCOMPAT_ENCRYPT		0x10000
+#define EXT4_FEATURE_INCOMPAT_FNAME_ENCODING	0x20000
 
 #define EXT4_FEATURE_COMPAT_FUNCS(name, flagname) \
 static inline bool ext4_has_feature_##name(struct super_block *sb) \
@@ -1749,6 +1752,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(csum_seed,		CSUM_SEED)
 EXT4_FEATURE_INCOMPAT_FUNCS(largedir,		LARGEDIR)
 EXT4_FEATURE_INCOMPAT_FUNCS(inline_data,	INLINE_DATA)
 EXT4_FEATURE_INCOMPAT_FUNCS(encrypt,		ENCRYPT)
+EXT4_FEATURE_INCOMPAT_FUNCS(fname_encoding,	FNAME_ENCODING)
 
 #define EXT2_FEATURE_COMPAT_SUPP	EXT4_FEATURE_COMPAT_EXT_ATTR
 #define EXT2_FEATURE_INCOMPAT_SUPP	(EXT4_FEATURE_INCOMPAT_FILETYPE| \
@@ -1776,6 +1780,7 @@ EXT4_FEATURE_INCOMPAT_FUNCS(encrypt,		ENCRYPT)
 					 EXT4_FEATURE_INCOMPAT_MMP | \
 					 EXT4_FEATURE_INCOMPAT_INLINE_DATA | \
 					 EXT4_FEATURE_INCOMPAT_ENCRYPT | \
+					 EXT4_FEATURE_INCOMPAT_FNAME_ENCODING | \
 					 EXT4_FEATURE_INCOMPAT_CSUM_SEED | \
 					 EXT4_FEATURE_INCOMPAT_LARGEDIR)
 #define EXT4_FEATURE_RO_COMPAT_SUPP	(EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \
Signature · 1 line
2.20.0.rc2
#20 lore raw

PATCH v4 20/23 ext4: Include encoding information in the superblock

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

Support for encoding is considered an incompatible feature, since it has potential to create collisions of file names in existing filesystems. If the feature flag is not enabled, the entire filesystem will operate on opaque byte sequences, respecting the original behavior.

The charset data is encoded in a new field in the superblock using a magic number specific to ext4. This is the easiest way I found to avoid writing the name of the charset in the superblock. The magic number is mapped to the exact NLS table, but the mapping is specific to ext4. Since we don't have any commitment to support old encodings, the only encodings I am supporting right now is utf8-11.0 and ascii, both using the NLS abstraction.

The current implementation prevents the user from enabling encoding and per-directory encryption on the same filesystem at the same time. The incompatibility between these features lies in how we do efficient directory searches when we cannot be sure the encryption of the user provided fname will match the actual hash stored in the disk without decrypting every directory entry, because of normalization cases. My quickest solution is to simply block the concurrent use of these features for now, and enable it later, once we have a better solution.

Changes since v2:
- Split superblock bitfield reservation into another patch.
- Rename s_ioencoding -> s_encoding
- Remove encoding_info from in-memory superblock.

Changes since v1:
- Guard code with CONFIG_NLS.
- Use 16 bits for s_ioencoding.
- Split mount option from this patch

---
 fs/ext4/ext4.h  |  7 +++++
 fs/ext4/super.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)
fs/ext4/ext4.h +7 -0
index 52c9e8b948a0..c21717a19106 100644
@@ -1338,6 +1338,9 @@ struct ext4_super_block {
 /* Number of quota types we support */
 #define EXT4_MAXQUOTAS 3
 
+#define EXT4_ENC_ASCII		0
+#define EXT4_ENC_UTF8_11_0	1
+
 /*
  * fourth extended-fs super-block data in memory
  */
@@ -1387,6 +1390,10 @@ struct ext4_sb_info {
 	struct kobject s_kobj;
 	struct completion s_kobj_unregister;
 	struct super_block *s_sb;
+#ifdef CONFIG_NLS
+	struct nls_table *s_encoding;
+	__u16 s_encoding_flags;
+#endif
 
 	/* Journaling */
 	struct journal_s *s_journal;
fs/ext4/super.c +77 -0

113 lines not on this page · raw message

Signature · 1 line
2.20.0.rc2
#21 lore raw

PATCH v4 21/23 ext4: Support encoding-aware file name lookups

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

This patch implements the actual support for encoding-aware file name lookups in ext4, based on the feature bit and the encoding stored in the superblock.

A filesystem that has the encoding feature set is able to find files even if the name used by userspace is not exactly the same, but if it is an equivalent string. This operation will be called and inexact-match name search.

Ext4 only stores the first equivalent name dentry used in the dcache. This is done to prevent unintentional duplication of dentries in the dcache, while also allowing the VFS code to quickly find the right entry in the cache despite what equivalent string was used without resorting to ->lookup().

d_hash() is implemented as the hash of the normalized string, such that we always have a well-known bucket for all the equivalencies of the same string. d_compare uses the nls_strncmp() infrastructure, which should handle the comparison of equivalent names as well. If the filesystem's normalization type is PLAIN, though, we can just reuse the VFS hash.

For now, negative lookups are not inserted in the dcache, since they would need to be invalidated anyway, because we can't trust missing file dentries. This is bad for performance but requires some leveraging of the vfs layer to fix. We can live without that for now, and so does everyone else.

DX is supported by modifying the hashes to make them encoding-aware. The new disk hashes are also calculated as the hash of the normalized string, instead of the string directly. This allows us to efficiently search for file names in the htree without requiring the user to provide the exact name.

Changes since v2:
- Don't use d_add_ci.
- Squash the dcache hooks into this patch.
- Rename sbi->encoding -> sbi->s_encoding.

Changes since v1:
- Support normalized htree hashes.
- Guard code with CONFIG_NLS.
- Use qstr->len instead of strlen in dcache hookups.

---
 fs/ext4/dir.c    | 45 ++++++++++++++++++++++++++++
 fs/ext4/ext4.h   | 12 ++++++--
 fs/ext4/hash.c   | 34 ++++++++++++++++++++-
 fs/ext4/ialloc.c |  2 +-
 fs/ext4/inline.c |  2 +-
 fs/ext4/namei.c  | 78 +++++++++++++++++++++++++++++++++++++++++-------
 fs/ext4/super.c  |  6 ++++
 7 files changed, 163 insertions(+), 16 deletions(-)
fs/ext4/dir.c +45 -0
index f93f9881ec18..efb75c204551 100644
@@ -26,6 +26,7 @@
 #include <linux/buffer_head.h>
 #include <linux/slab.h>
 #include <linux/iversion.h>
+#include <linux/nls.h>
 #include "ext4.h"
 #include "xattr.h"
 
@@ -662,3 +663,47 @@ const struct file_operations ext4_dir_operations = {
 	.open		= ext4_dir_open,
 	.release	= ext4_release_dir,
 };
+
+#ifdef CONFIG_NLS
+static int ext4_d_compare(const struct dentry *dentry, unsigned int len,
+			  const char *str, const struct qstr *name)
+{
+	struct nls_table *charset = EXT4_SB(dentry->d_sb)->s_encoding;
+
+	return nls_strncmp(charset, str, len, name->name, name->len);
+}
+
+static int ext4_d_hash(const struct dentry *dentry, struct qstr *q)
+{
+	const struct nls_table *charset = EXT4_SB(dentry->d_sb)->s_encoding;
+	unsigned char *norm;
+	int len, ret = 0;
+
+	/* If normalization is TYPE_PLAIN, we can just reuse the vfs
+	 * hash. */
+	if (IS_NORMALIZATION_TYPE_ALL_PLAIN(charset))
+	    return 0;
+
+	norm = kmalloc(PATH_MAX, GFP_ATOMIC);
+	if (!norm)
+		return -ENOMEM;
+
+	len = nls_normalize(charset, q->name, q->len, norm, PATH_MAX);
+
+	if (len < 0) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	q->hash = full_name_hash(dentry, norm, len);
+
+out:
+	kfree (norm);
+	return ret;
+}
+
+const struct dentry_operations ext4_dentry_ops = {
+	.d_hash = ext4_d_hash,
+	.d_compare = ext4_d_compare,
+};
+#endif
fs/ext4/ext4.h +10 -2
index c21717a19106..e84a6605a19a 100644
@@ -1341,6 +1341,11 @@ struct ext4_super_block {
 #define EXT4_ENC_ASCII		0
 #define EXT4_ENC_UTF8_11_0	1
 
+/*
+ * Flags for ext4_sb_info.s_encoding_flags.  Be careful when modifying
+ * these, as they must match their NLS counterpart. */
+#define EXT4_ENC_STRICT_MODE_FL	(1 << 0)
+
 /*
  * fourth extended-fs super-block data in memory
  */
@@ -2387,8 +2392,8 @@ extern int ext4_check_all_de(struct inode *dir, struct buffer_head *bh,
 extern int ext4_sync_file(struct file *, loff_t, loff_t, int);
 
 /* hash.c */
-extern int ext4fs_dirhash(const char *name, int len, struct
-			  dx_hash_info *hinfo);
+extern int ext4fs_dirhash(const struct inode *dir, const char *name, int len,
+			  struct dx_hash_info *hinfo);
 
 /* ialloc.c */
 extern struct inode *__ext4_new_inode(handle_t *, struct inode *, umode_t,
@@ -2971,6 +2976,9 @@ static inline void ext4_unlock_group(struct super_block *sb,
 
 /* dir.c */
 extern const struct file_operations ext4_dir_operations;
+#ifdef CONFIG_NLS
+extern const struct dentry_operations ext4_dentry_ops;
+#endif
 
 /* file.c */
 extern const struct inode_operations ext4_file_inode_operations;
fs/ext4/hash.c +33 -1
index e22dcfab308b..8ec9c7145987 100644
@@ -6,6 +6,7 @@
  */
 
 #include <linux/fs.h>
+#include <linux/nls.h>
 #include <linux/compiler.h>
 #include <linux/bitops.h>
 #include "ext4.h"
@@ -196,7 +197,8 @@ static void str2hashbuf_unsigned(const char *msg, int len, __u32 *buf, int num)
  * represented, and whether or not the returned hash is 32 bits or 64
  * bits.  32 bit hashes will return 0 for the minor hash.
  */
-int ext4fs_dirhash(const char *name, int len, struct dx_hash_info *hinfo)
+static int __ext4fs_dirhash(const char *name, int len,
+			    struct dx_hash_info *hinfo)
 {
 	__u32	hash;
 	__u32	minor_hash = 0;
@@ -266,3 +268,33 @@ int ext4fs_dirhash(const char *name, int len, struct dx_hash_info *hinfo)
 	hinfo->minor_hash = minor_hash;
 	return 0;
 }
+
+int ext4fs_dirhash(const struct inode *dir, const char *name, int len,
+		   struct dx_hash_info *hinfo)
+{
+#ifdef CONFIG_NLS
+	const struct nls_table *charset = EXT4_SB(dir->i_sb)->s_encoding;
+	int r, dlen;
+	unsigned char *buff;
+
+	if (len && charset) {
+		buff = kzalloc(sizeof (char) * PATH_MAX, GFP_KERNEL);
+		if (!buff)
+			return -1;
+
+		dlen = nls_normalize(charset, name, len, buff, PATH_MAX);
+
+		if (dlen < 0) {
+			kfree(buff);
+			goto opaque_seq;
+		}
+
+		r = __ext4fs_dirhash(buff, dlen, hinfo);
+
+		kfree(buff);
+		return r;
+	}
+opaque_seq:
+#endif
+	return __ext4fs_dirhash(name, len, hinfo);
+}
fs/ext4/ialloc.c +1 -1
index 014f6a698cb7..1ef355549abf 100644
@@ -455,7 +455,7 @@ static int find_group_orlov(struct super_block *sb, struct inode *parent,
 		if (qstr) {
 			hinfo.hash_version = DX_HASH_HALF_MD4;
 			hinfo.seed = sbi->s_hash_seed;
-			ext4fs_dirhash(qstr->name, qstr->len, &hinfo);
+			ext4fs_dirhash(parent, qstr->name, qstr->len, &hinfo);
 			grp = hinfo.hash;
 		} else
 			grp = prandom_u32();
fs/ext4/inline.c +1 -1
index 9c4bac18cc6c..10b9d3dcec4e 100644
@@ -1404,7 +1404,7 @@ int htree_inlinedir_to_tree(struct file *dir_file,
 			}
 		}
 
-		ext4fs_dirhash(de->name, de->name_len, hinfo);
+		ext4fs_dirhash(dir, de->name, de->name_len, hinfo);
 		if ((hinfo->hash < start_hash) ||
 		    ((hinfo->hash == start_hash) &&
 		     (hinfo->minor_hash < start_minor_hash)))
fs/ext4/namei.c +67 -11

187 lines not on this page · raw message

fs/ext4/super.c +6 -0
index e64a9ed2ca12..5ac1ed77f36b 100644
@@ -4412,6 +4412,12 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
 		iput(root);
 		goto failed_mount4;
 	}
+
+#ifdef CONFIG_NLS
+	if (sbi->s_encoding)
+		sb->s_d_op = &ext4_dentry_ops;
+#endif
+
 	sb->s_root = d_make_root(root);
 	if (!sb->s_root) {
 		ext4_msg(sb, KERN_ERR, "get root dentry failed");
Signature · 1 line
2.20.0.rc2
#22 lore raw

PATCH v4 22/23 ext4: Implement EXT4_CASEFOLD_FL flag

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

Casefold is a flag applied to directories and inherited by its children which states that the directory requires case-insensitive searches. This flag can only be enabled on empty directories for filesystems that support the encoding feature, thus preventing collision of file names that only differ by case.

Enconding-awareness is also required because we consider the casefold operation not be defined for opaque byte sequences.

Changes since v2:
- Rename sbi->encoding -> sbi->s_encoding.

Changes since v1:
- Moved the CASEFOLD_FL to prevent collision with reserved verity flag.

---
 fs/ext4/dir.c      | 30 ++++++++++++++++++++++--------
 fs/ext4/ext4.h     |  7 ++++---
 fs/ext4/hash.c     |  6 +++++-
 fs/ext4/inode.c    |  4 +++-
 fs/ext4/ioctl.c    | 18 ++++++++++++++++++
 fs/ext4/namei.c    | 13 ++++++++++---
 include/linux/fs.h |  2 ++
 7 files changed, 64 insertions(+), 16 deletions(-)
fs/ext4/dir.c +22 -8
index efb75c204551..43b91747f7e7 100644
@@ -670,6 +670,10 @@ static int ext4_d_compare(const struct dentry *dentry, unsigned int len,
 {
 	struct nls_table *charset = EXT4_SB(dentry->d_sb)->s_encoding;
 
+	if (IS_CASEFOLDED(dentry->d_parent->d_inode))
+		return nls_strncasecmp(charset, str, len, name->name,
+				       name->len);
+
 	return nls_strncmp(charset, str, len, name->name, name->len);
 }
 
@@ -679,16 +683,26 @@ static int ext4_d_hash(const struct dentry *dentry, struct qstr *q)
 	unsigned char *norm;
 	int len, ret = 0;
 
-	/* If normalization is TYPE_PLAIN, we can just reuse the vfs
-	 * hash. */
-	if (IS_NORMALIZATION_TYPE_ALL_PLAIN(charset))
-	    return 0;
+	if (!IS_CASEFOLDED(dentry->d_inode)) {
 
-	norm = kmalloc(PATH_MAX, GFP_ATOMIC);
-	if (!norm)
-		return -ENOMEM;
+		/* If normalization is TYPE_PLAIN, we can just reuse the
+		 * VFS hash.
+		 */
+		if (IS_NORMALIZATION_TYPE_ALL_PLAIN(charset))
+			return 0;
 
-	len = nls_normalize(charset, q->name, q->len, norm, PATH_MAX);
+		norm = kmalloc(PATH_MAX, GFP_ATOMIC);
+		if (!norm)
+			return -ENOMEM;
+
+		len = nls_normalize(charset, q->name, q->len, norm, PATH_MAX);
+	} else {
+		norm = kmalloc(PATH_MAX, GFP_ATOMIC);
+		if (!norm)
+			return -ENOMEM;
+
+		len = nls_casefold(charset, q->name, q->len, norm, PATH_MAX);
+	}
 
 	if (len < 0) {
 		ret = -EINVAL;
fs/ext4/ext4.h +4 -3
index e84a6605a19a..d21ed5e88302 100644
@@ -400,10 +400,11 @@ struct flex_groups {
 #define EXT4_EOFBLOCKS_FL		0x00400000 /* Blocks allocated beyond EOF */
 #define EXT4_INLINE_DATA_FL		0x10000000 /* Inode has inline data. */
 #define EXT4_PROJINHERIT_FL		0x20000000 /* Create with parents projid */
+#define EXT4_CASEFOLD_FL		0x40000000 /* Casefolded file */
 #define EXT4_RESERVED_FL		0x80000000 /* reserved for ext4 lib */
 
-#define EXT4_FL_USER_VISIBLE		0x304BDFFF /* User visible flags */
-#define EXT4_FL_USER_MODIFIABLE		0x204BC0FF /* User modifiable flags */
+#define EXT4_FL_USER_VISIBLE		0x704BDFFF /* User visible flags */
+#define EXT4_FL_USER_MODIFIABLE		0x604BC0FF /* User modifiable flags */
 
 /* Flags we can manipulate with through EXT4_IOC_FSSETXATTR */
 #define EXT4_FL_XFLAG_VISIBLE		(EXT4_SYNC_FL | \
@@ -418,7 +419,7 @@ struct flex_groups {
 			   EXT4_SYNC_FL | EXT4_NODUMP_FL | EXT4_NOATIME_FL |\
 			   EXT4_NOCOMPR_FL | EXT4_JOURNAL_DATA_FL |\
 			   EXT4_NOTAIL_FL | EXT4_DIRSYNC_FL |\
-			   EXT4_PROJINHERIT_FL)
+			   EXT4_PROJINHERIT_FL | EXT4_CASEFOLD_FL)
 
 /* Flags that are appropriate for regular files (all but dir-specific ones). */
 #define EXT4_REG_FLMASK (~(EXT4_DIRSYNC_FL | EXT4_TOPDIR_FL))
fs/ext4/hash.c +5 -1
index 8ec9c7145987..78cb97664a33 100644
@@ -282,7 +282,11 @@ int ext4fs_dirhash(const struct inode *dir, const char *name, int len,
 		if (!buff)
 			return -1;
 
-		dlen = nls_normalize(charset, name, len, buff, PATH_MAX);
+		if (!IS_CASEFOLDED(dir))
+			dlen = nls_normalize(charset, name, len, buff,
+					     PATH_MAX);
+		else
+			dlen = nls_casefold(charset, name, len, buff, PATH_MAX);
 
 		if (dlen < 0) {
 			kfree(buff);
fs/ext4/inode.c +3 -1
index 22a9d8159720..9908d7d98b6e 100644
@@ -4745,9 +4745,11 @@ void ext4_set_inode_flags(struct inode *inode)
 		new_fl |= S_DAX;
 	if (flags & EXT4_ENCRYPT_FL)
 		new_fl |= S_ENCRYPTED;
+	if (flags & EXT4_CASEFOLD_FL)
+		new_fl |= S_CASEFOLD;
 	inode_set_flags(inode, new_fl,
 			S_SYNC|S_APPEND|S_IMMUTABLE|S_NOATIME|S_DIRSYNC|S_DAX|
-			S_ENCRYPTED);
+			S_ENCRYPTED|S_CASEFOLD);
 }
 
 static blkcnt_t ext4_inode_blocks(struct ext4_inode *raw_inode,
fs/ext4/ioctl.c +18 -0
index 0edee31913d1..ef4ffe681836 100644
@@ -231,6 +231,7 @@ static int ext4_ioctl_setflags(struct inode *inode,
 	struct ext4_iloc iloc;
 	unsigned int oldflags, mask, i;
 	unsigned int jflag;
+	struct super_block *sb = inode->i_sb;
 
 	/* Is it quota file? Do not allow user to mess with it */
 	if (ext4_is_quota_file(inode))
@@ -275,6 +276,23 @@ static int ext4_ioctl_setflags(struct inode *inode,
 			goto flags_out;
 	}
 
+	if ((flags ^ oldflags) & EXT4_CASEFOLD_FL) {
+		if (!ext4_has_feature_fname_encoding(sb)) {
+			err = -EOPNOTSUPP;
+			goto flags_out;
+		}
+
+		if (!S_ISDIR(inode->i_mode)) {
+			err = -ENOTDIR;
+			goto flags_out;
+		}
+
+		if (!ext4_empty_dir(inode)) {
+			err = -ENOTEMPTY;
+			goto flags_out;
+		}
+	}
+
 	handle = ext4_journal_start(inode, EXT4_HT_INODE, 1);
 	if (IS_ERR(handle)) {
 		err = PTR_ERR(handle);
fs/ext4/namei.c +10 -3
index 23e0e911b3fe..a21f0d7227db 100644
@@ -1278,9 +1278,16 @@ static inline bool ext4_match(const struct inode *parent,
 
 #ifdef CONFIG_NLS
 	if (sbi->s_encoding) {
-		return !nls_strncmp(sbi->s_encoding,
-				    de->name, de->name_len,
-				    f.disk_name.name, f.disk_name.len);
+		if (!IS_CASEFOLDED(parent))
+			return !nls_strncmp(sbi->s_encoding,
+					    de->name, de->name_len,
+					    fname->disk_name.name,
+					    fname->disk_name.len);
+		else
+			return !nls_strncasecmp(sbi->s_encoding,
+						de->name, de->name_len,
+						fname->disk_name.name,
+						fname->disk_name.len);
 	}
 #endif
 
include/linux/fs.h +2 -0
index c95c0807471f..69abaca207c0 100644
@@ -1947,6 +1947,7 @@ struct super_operations {
 #define S_DAX		0	/* Make all the DAX code disappear */
 #endif
 #define S_ENCRYPTED	16384	/* Encrypted file (using fs/crypto/) */
+#define S_CASEFOLD	32768	/* Casefolded file */
 
 /*
  * Note that nosuid etc flags are inode-specific: setting some file-system
@@ -1987,6 +1988,7 @@ static inline bool sb_rdonly(const struct super_block *sb) { return sb->s_flags
 #define IS_NOSEC(inode)		((inode)->i_flags & S_NOSEC)
 #define IS_DAX(inode)		((inode)->i_flags & S_DAX)
 #define IS_ENCRYPTED(inode)	((inode)->i_flags & S_ENCRYPTED)
+#define IS_CASEFOLDED(inode)	((inode)->i_flags & S_CASEFOLD)
 
 #define IS_WHITEOUT(inode)	(S_ISCHR(inode->i_mode) && \
 				 (inode)->i_rdev == WHITEOUT_DEV)
Signature · 1 line
2.20.0.rc2
#23 lore raw

PATCH v4 23/23 docs: ext4.rst: Document encoding and case-insensitive

In reply to Gabriel Krisman Bertazi

From: Gabriel Krisman Bertazi <[email protected]>

Introduces the encoding-awareness and case-insensitive features on ext4 for system administrators. Explain the minimum of design decisions that are important for sysadmins enabling this feature.

---
 Documentation/admin-guide/ext4.rst | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
Documentation/admin-guide/ext4.rst +29 -0
index e506d3dae510..f42c682acecc 100644
@@ -91,10 +91,39 @@ Currently Available
 * large block (up to pagesize) support
 * efficient new ordered mode in JBD2 and ext4 (avoid using buffer head to force
   the ordering)
+* Encoding aware file names
+* Case insensitive file name lookups
 
 [1] Filesystems with a block size of 1k may see a limit imposed by the
 directory hash tree having a maximum depth of two.
 
+Encoding-aware file names and case-insensitive lookups
+======================================================
+
+Ext4 optionally supports filesystem-wide charset knowledge when handling
+file names, which allows the user to perform file system lookups using
+charset equivalent versions of the same file name, and optionally ensure
+that no invalid names are held by the filesystem.  charset encoding
+awareness is also essential for performing case-insensitive lookups,
+because it is what defines the casefold operation.
+
+The case-insensitive file name lookup feature is supported in a smaller
+granularity, on a per-directory basis, allowing the user to mix
+case-insensitive and case-sensitive directories in the same filesystem.
+It is enabled by flipping a file attribute on an empty directory.  For
+the reason stated above, the filesystem must have encoding enabled to
+use this feature.
+
+When we change from filenames as opaque byte sequences to seeing them as
+encoded strings we need to address what happens when a program tries to
+create a file with an invalid name.  The Natural Language System within
+the kernel leaves the decision of what to do in this case to the
+filesystem, which select its preferred behavior by enabling/disabling
+the strict mode in NLS.  When Ext4 encounters one of those strings, it
+falls back to considering the entire string as an opaque byte sequence,
+which still allows the user to operate on that file but the
+case-insensitive and equivalent sequence lookups won't work.
+
 Options
 =======
 
Signature · 1 line
2.20.0.rc2
reviewer
#24 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 10 lines · Gabriel Krisman Bertazi

Hi,

[Resending to include fsdevel, as requested by Dave Chinner]

Following the e2fsprogs changes, these are the corresponding kernel-side modifications to support the fname_encoding feature.

The patches are split in two parts. The fist 14 patches are refactoring and improvements to the NLS code, including the utf8 normalization support. The final patches implement the fname_encoding feature in ext4.

Hi,

Please include some justification and use case(s) in the patch description.

Thanks.

Quoted 150 lines

To test this feature, you need to use the tip of e2fsprogs branch, which already include support for enabling this feature.

As usual, the ucd files are not included in this email because they are too large, and would actually cause the email message to bounce.

There are two test files for this in a private xfstests branch, that I plan to submit upstream once we get this series merged:

https://gitlab.collabora.com/krisman/xfstests.git -b encoding_v4

I also tested this with the xfstests smoke tests using two scenarios:
(1) a non-encoding TEST_DEV; (2) a utf8 enabled TEST_DEV. On both cases, no unrelated regressions where observed. With my branch of xfstests above, that fixes some related tests, I didn't observe any regressions.

Gabriel Krisman Bertazi (19):
nls: Wrap uni2char/char2uni callers
nls: Wrap charset field access
nls: Wrap charset hooks in ops structure
nls: Split default charset from NLS core
nls: Split struct nls_charset from struct nls_table
nls: Add support for multiple versions of an encoding
nls: Implement NLS_STRICT_MODE flag
nls: Let charsets define the behavior of tolower/toupper
nls: Add new interface for string comparisons
nls: Add optional normalization and casefold hooks
nls: ascii: Support validation and normalization operations
nls: utf8: Move nls-utf8{,-core}.c
nls: utf8: Integrate utf8 normalization code with utf8 charset
nls: utf8: Introduce test module for normalized utf8 implementation
ext4: Reserve superblock fields for encoding information
ext4: Include encoding information in the superblock
ext4: Support encoding-aware file name lookups
ext4: Implement EXT4_CASEFOLD_FL flag
docs: ext4.rst: Document encoding and case-insensitive

Olaf Weber (4):
nls: utf8: Add unicode character database files
scripts: add trie generator for UTF-8
nls: utf8: Introduce code for UTF-8 normalization
nls: utf8n: reduce the size of utf8data[]

 Documentation/admin-guide/ext4.rst   |   29 +
 fs/befs/linuxvfs.c                   |    8 +-
 fs/cifs/cifs_unicode.c               |   15 +-
 fs/cifs/cifsfs.c                     |    2 +-
 fs/cifs/connect.c                    |    2 +-
 fs/cifs/dir.c                        |    7 +-
 fs/ext4/dir.c                        |   59 +
 fs/ext4/ext4.h                       |   33 +-
 fs/ext4/hash.c                       |   38 +-
 fs/ext4/ialloc.c                     |    2 +-
 fs/ext4/inline.c                     |    2 +-
 fs/ext4/inode.c                      |    4 +-
 fs/ext4/ioctl.c                      |   18 +
 fs/ext4/namei.c                      |   85 +-
 fs/ext4/super.c                      |   83 +
 fs/fat/dir.c                         |   13 +-
 fs/fat/inode.c                       |    6 +-
 fs/fat/namei_vfat.c                  |    6 +-
 fs/hfs/super.c                       |    6 +-
 fs/hfs/trans.c                       |    9 +-
 fs/hfsplus/options.c                 |    2 +-
 fs/hfsplus/unicode.c                 |    6 +-
 fs/isofs/inode.c                     |    5 +-
 fs/isofs/joliet.c                    |    3 +-
 fs/jfs/jfs_unicode.c                 |    9 +-
 fs/jfs/super.c                       |    3 +-
 fs/nls/Kconfig                       |   15 +
 fs/nls/Makefile                      |   20 +
 fs/nls/mac-celtic.c                  |   34 +-
 fs/nls/mac-centeuro.c                |   34 +-
 fs/nls/mac-croatian.c                |   34 +-
 fs/nls/mac-cyrillic.c                |   34 +-
 fs/nls/mac-gaelic.c                  |   34 +-
 fs/nls/mac-greek.c                   |   34 +-
 fs/nls/mac-iceland.c                 |   34 +-
 fs/nls/mac-inuit.c                   |   34 +-
 fs/nls/mac-roman.c                   |   34 +-
 fs/nls/mac-romanian.c                |   34 +-
 fs/nls/mac-turkish.c                 |   34 +-
 fs/nls/nls_ascii.c                   |   84 +-
 fs/nls/nls_core.c                    |  163 ++
 fs/nls/nls_cp1250.c                  |   34 +-
 fs/nls/nls_cp1251.c                  |   34 +-
 fs/nls/nls_cp1255.c                  |   36 +-
 fs/nls/nls_cp437.c                   |   34 +-
 fs/nls/nls_cp737.c                   |   34 +-
 fs/nls/nls_cp775.c                   |   34 +-
 fs/nls/nls_cp850.c                   |   34 +-
 fs/nls/nls_cp852.c                   |   34 +-
 fs/nls/nls_cp855.c                   |   34 +-
 fs/nls/nls_cp857.c                   |   34 +-
 fs/nls/nls_cp860.c                   |   34 +-
 fs/nls/nls_cp861.c                   |   34 +-
 fs/nls/nls_cp862.c                   |   34 +-
 fs/nls/nls_cp863.c                   |   34 +-
 fs/nls/nls_cp864.c                   |   34 +-
 fs/nls/nls_cp865.c                   |   34 +-
 fs/nls/nls_cp866.c                   |   34 +-
 fs/nls/nls_cp869.c                   |   34 +-
 fs/nls/nls_cp874.c                   |   36 +-
 fs/nls/nls_cp932.c                   |   36 +-
 fs/nls/nls_cp936.c                   |   36 +-
 fs/nls/nls_cp949.c                   |   36 +-
 fs/nls/nls_cp950.c                   |   36 +-
 fs/nls/{nls_base.c => nls_default.c} |  124 +-
 fs/nls/nls_euc-jp.c                  |   29 +-
 fs/nls/nls_iso8859-1.c               |   34 +-
 fs/nls/nls_iso8859-13.c              |   34 +-
 fs/nls/nls_iso8859-14.c              |   34 +-
 fs/nls/nls_iso8859-15.c              |   34 +-
 fs/nls/nls_iso8859-2.c               |   34 +-
 fs/nls/nls_iso8859-3.c               |   34 +-
 fs/nls/nls_iso8859-4.c               |   34 +-
 fs/nls/nls_iso8859-5.c               |   34 +-
 fs/nls/nls_iso8859-6.c               |   34 +-
 fs/nls/nls_iso8859-7.c               |   34 +-
 fs/nls/nls_iso8859-9.c               |   34 +-
 fs/nls/nls_koi8-r.c                  |   34 +-
 fs/nls/nls_koi8-ru.c                 |   30 +-
 fs/nls/nls_koi8-u.c                  |   34 +-
 fs/nls/nls_utf8-core.c               |  328 +++
 fs/nls/nls_utf8-norm.c               |  797 ++++++
 fs/nls/nls_utf8-selftest.c           |  316 +++
 fs/nls/nls_utf8.c                    |   67 -
 fs/nls/ucd/README                    |   34 +
 fs/nls/utf8n.h                       |  117 +
 fs/ntfs/inode.c                      |    2 +-
 fs/ntfs/super.c                      |    6 +-
 fs/ntfs/unistr.c                     |   13 +-
 fs/udf/super.c                       |    3 +-
 fs/udf/unicode.c                     |    4 +-
 include/linux/fs.h                   |    2 +
 include/linux/nls.h                  |  293 ++-
 scripts/Makefile                     |    1 +
 scripts/mkutf8data.c                 | 3392 ++++++++++++++++++++++++++
 95 files changed, 7287 insertions(+), 618 deletions(-)
 create mode 100644 fs/nls/nls_core.c
 rename fs/nls/{nls_base.c => nls_default.c} (89%)
 create mode 100644 fs/nls/nls_utf8-core.c
 create mode 100644 fs/nls/nls_utf8-norm.c
 create mode 100644 fs/nls/nls_utf8-selftest.c
 delete mode 100644 fs/nls/nls_utf8.c
 create mode 100644 fs/nls/ucd/README
 create mode 100644 fs/nls/utf8n.h
 create mode 100644 scripts/mkutf8data.c
Signature · 1 line
~Randy

A message that is not in the archive (sent off-list or not yet received): [email protected]

#26 lore raw
Quoted 6 lines · Theodore Y. Ts'o

There's a patch series that's been baking for a while that will likely go upstream either in the next upcoming merge window, or the one after that. Since it adds support for Unicode case-folding, it involves a non-trivial number of changes to fs/nls. As near as I can tell, no one is really maintaining fs/nls.

Christ.

Why do people want to do this? We know it's a crazy and stupid thing to do. And we know that, exactly because people have done it, and it has always been a mistake.

It causes actual and very subtle security issues.

It breaks things subtly even when they supposedly "know" about case folding because different things will do it differently (ie user space vs kernel space not having the *exact* same rules due to using different tables, for example).

It doesn't work with locales, because people often want different locales at the same time.

And it slows things down enormously because you can't do hashing well, and comparisons get hugely more expensive.

And to add insult to injury, people always implement it so *horribly* badly that it's not even funny.

For example, the usual way that people do it is to case-fold two strings, and then compare the end results. And that's *incredibly* stupid and slow and generates extra temporary allocations etc.

Or people to it character-by-character instead, and don't understand utf-8 (which is literally designed to be easy to see character boundaries *without* having to do a full decode!), and do *that* incredibly badly instead.

And when you create a file with an ambiguous name, what does readdir report? Does it report the name you used, some normalized thing, or what?

Finally, people then invariably do it in ways that preclude any concurrent sane uses.

For example, they make it a single mount-time flag for the whole filesystem, so now if you are (for example) wanting to do emulation of bad system decisions, you now force the *host* to buy into the whole mistake too.

And they make it a whole-filesystem flag, instead of (for example) allowing just the emulated environment to do case-insensitive filesystem operations on an operation-by-operation basis, and possibly only within a particular subdirectory structure (or bind mount).

So the first thing I want to know is who really needs it, *why* they need it, and what the design is for.

Because I can almost guarantee that the design is horrible, and the reasons are really really bad.

And what *are* the case insensitivity rules, and how do you co-exist when there are two *different* folding rules at the same time? For example, OS X has some truly horrendously bad rules, that take the badness that Windows did to a whole different level. What if you're a file server (or emulation environment) and you want to expose the same filesystem to both of those environments?

Because it would quite possibly be a whole lot better to allow per-operation flags, so that you can do

    fd = openat(dir, path, O_RDONLY | O_ICASE);

so that you can allow *one* process to treat a filesystem as if it was case insensitive (think "Wine in with a ~/.wine/C directory"), without forcing the whole filesystem to be icase.

Yes, allowing concurrent use then generates whole new "interesting" questions, like "what happens if a case _sensitive_ user creates two files with names that are identical to a in-sensitive user", but they aren't necessarily any worse than the issues you face *not* allowing that.

Quoted 6 lines

Given your recent comments about not wanting to see pull requests for things outside of fs/xfs as part of the xfs pull, do you have any opinions about how to do manage this feature going upstream? My original plan was to send them through the ext4 tree, since I very much doubt Al cares much about nls issues, and they will only impact ext4.

I really want to know what is driving this insanity, and what the actual use-case is.

You have a diffstat, but not a git tree to look at what the heck is going on.

Seriously, case insensitivity is *such* a horrendously bad idea that people need to think about it deeply, and nobody seems to ever do that.

And yes, we have d_hash() and some rudimentary support for it in the VFS layer, but that VFS layer bit was always meant purely for interoperability filesystems that nobody really cared about as a real filesystem for Linux. Notably FAT and its ilk.

If we have a major native filesystem doing it, I think we need to actively think about the big picture and do it *right*. None of the crazy "ok, you can't even look things up in the dcache directly at all" stuff that we have as a hack to just allow _bad_ filesystems to do their thing.

So I think this is a bigger deal than that diffstat of yours implies. I don't think people understand just how *bad* case insensitivity is.

The old DOS/Mac people thought case insensitivity was a "helpful" idea, and that was understandable - but wrong - even back in the 80's. They are still living with the end result of that horrendously bad decision decades later. They've _tried_ to fix their bad decisions, and have never been able to (except, apparently, in iOS where somebody finally had a glimmer of a clue).

Linus

#27 lore raw
Quoted 6 lines · Linus Torvalds

Yes, allowing concurrent use then generates whole new "interesting" questions, like "what happens if a case _sensitive_ user creates two files with names that are identical to a in-sensitive user", but they aren't necessarily any worse than the issues you face *not* allowing that.

I'm hoping you are at least doing it per-directory. That makes at least the "oh, the whole filesystem needs to do this wrong" issue a bit less bad.

Just looking at the shortlog you posted, my guess is that the ext4 patches didn't even get *that* right, though. That shortlog "encoding information in superblock" implies this is the same kind of just horribly bad mess that we've seen before.

I really despise every single case-sensitive filesystem I have ever seen, exactly because nobody apparently spends even a minimal amount of effort on getting any of the basics remotely right. Every single case I've seen has been a huge nasty hack, with seriously bad system-wide consequences.

Linus

#28 lore raw
Quoted 4 lines · Linus Torvalds

I'm hoping you are at least doing it per-directory. That makes at least the "oh, the whole filesystem needs to do this wrong" issue a bit less bad.

So for example, if you do it per-directory, the rules could be something like:

- new directories (ie "mkdir()") inherit the icase/folding semantics
of the parent directory

- empty directories can have their case/folding rules changed with
some well-defined interface

and even from just those simple rules, now some icase behavior could be useful to testing.

Not just filesystem testing (although that would be a thing - thing fsstress), but for doing app development in a test directory.

Apps like git (and GNU fileutils) could use it for having test suites for FAT etc filesystems.

And cross-platform apps could use it as a "I want to check that I do the right thing" if you do development on Linux, but might have a portable app for other platforms.

If the whole filesystem is that way, nobody is going to do it. Sure, they could do it on a FAT filesystem using a USB disk, but nobody really does that. But if you can troivially just run your tests in a test subdirectory, it's another thing entirely.

So this is the kind of thing I mean when I think icase behavior for a major Linux filesystem should have a real _design_. It's really quite fundamentally different from the "oh, I need FAT to be icase" hack that we have now.

(We might also be able to make the dcache better at handling well-defined icase/folding rules, as opposed to the current "just give up, let the filesystem hash it" behavior).

Linus

#29 lore raw
Quoted 34 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds

I'm hoping you are at least doing it per-directory. That makes at least the "oh, the whole filesystem needs to do this wrong" issue a bit less bad.

So for example, if you do it per-directory, the rules could be something like:

- new directories (ie "mkdir()") inherit the icase/folding semantics of the parent directory

- empty directories can have their case/folding rules changed with some well-defined interface

and even from just those simple rules, now some icase behavior could be useful to testing.

Not just filesystem testing (although that would be a thing - thing fsstress), but for doing app development in a test directory.

Apps like git (and GNU fileutils) could use it for having test suites for FAT etc filesystems.

And cross-platform apps could use it as a "I want to check that I do the right thing" if you do development on Linux, but might have a portable app for other platforms.

If the whole filesystem is that way, nobody is going to do it. Sure, they could do it on a FAT filesystem using a USB disk, but nobody really does that. But if you can troivially just run your tests in a test subdirectory, it's another thing entirely.

Quoted 8 lines

So this is the kind of thing I mean when I think icase behavior for a major Linux filesystem should have a real _design_. It's really quite fundamentally different from the "oh, I need FAT to be icase" hack that we have now.

(We might also be able to make the dcache better at handling well-defined icase/folding rules, as opposed to the current "just give up, let the filesystem hash it" behavior).

In theory, we could store the encoding on a per-entry basis if we wanted, using the dir_data feature (this would consume 2-3 bytes per entry, depending on how rich an encoding type we wanted). The tricky part is how does the kernel know what the filename encoding is? How do we communicate the encoding type back to userspace?

Cheers, Andreas

A message that is not in the archive (sent off-list or not yet received): [email protected]

#31 lore raw
Quoted 8 lines · Theodore Y. Ts'o

Whether or not case-folding is being done is per-directory (it's a flag on the directory set by chattr) . What encoding is supported (and we only will support two, ASCII and UTF-8) is per-file system. I personally believe it's insane to try to encode a large number of encodings, like big5, or iso-8859-1, etc. on a per-directory basis. Either don't do encodings at all, or use utf-8. Period. I believe
you made a similar request for git metadata, no? :-)

Absolutely.

But if you only support ascii or utf-8, then why are you messing with the nls part? That makes no sense.

You can't have it both ways.

Either you have a horrible fundamental design mistake that has different per-filesystem locales, or you don't.

If you don't, you shouldn't be touching any of the nls code.

Whatever unicode tables you use for case folding shouldn't be in the nls code.

Linus

maintainer
#32 lore raw
Quoted 11 lines · Linus Torvalds

But if you only support ascii or utf-8, then why are you messing with the nls part? That makes no sense.

You can't have it both ways.

Either you have a horrible fundamental design mistake that has different per-filesystem locales, or you don't.

If you don't, you shouldn't be touching any of the nls code.

Whatever unicode tables you use for case folding shouldn't be in the nls code.

Gabriel added the Unicode tables for case folding to the fs/nls directory. If you'd prefer that we put them somewhere else, we can; do you have a preference?

- Ted

#33 lore raw
Quoted 4 lines · Theodore Y. Ts'o

Gabriel added the Unicode tables for case folding to the fs/nls directory. If you'd prefer that we put them somewhere else, we can; do you have a preference?

I have a really hard time judging, since I haven't seen the code, just a random diffstat and shortlog.

First off, there is no such thing as "one" unicode table for case folding. There are lots and lots of tables, and I'm not clear what table it is all about.

For example, both OS X and Windows do some form of case folding on unicode. They don't do the *same* folding, though.

There are also various locale variations to case folding. This is where I thought your nls choice came from, but then you tried to imply that there are no locale issues and that directories can just have a single flag to enable/disable the folding.

In some locales, "SS" and "ß" (perhaps "SZ" too) will compare the same in case-insensitivity. Crazy in general, and afaik modern unicode even has a real upper-case "ß" so it's arguably legacy, but...

And that's all entirely independent of the issues with all the combining characters, modifier letters, white-space, overlong utf8 questions, etc etc.

It's also easy to generate overlong utf-8 that decodes to '/', for example. Some broken systems might consider that identical to a real '/' and it matters for path lookup.

So what's the actual code? What rules did you happen to pick? Did you take the windows rules as-is (I _think_ they may be documented) since the primary target apparently is just samba performance?

And even if the answer is "we follow NTFS rules", which *version* of NTFS folding rules are you using if you're trying to speed up samba, for example? Because afaik they have changed over time.

Is the *only* target samba? You are never interested for local loads like "oh, people want to run Wine and might need it" or the application testing parts?

All of these matter.

For example, if it's some "ext4 special case just for samba", then perhaps the logical place to put all this is just in fs/ext4/ and not bother anybody else about it.

But if it might be useful as some generic "NTFS hashing" library, then make it that.

Linus

maintainer
#34 lore raw
Quoted 6 lines · Linus Torvalds

First off, there is no such thing as "one" unicode table for case folding. There are lots and lots of tables, and I'm not clear what table it is all about.

For example, both OS X and Windows do some form of case folding on unicode. They don't do the *same* folding, though.

So things are much better in recent years. In the past it was kind of a disaster, but the world is converging enough that the latest versions of Mac OS'x APFS and Windows NTFS behave pretty much the same way. They are both case-insensitive, case-preserving and normalization-preserving, normalization-insensitive with respect to filenames.

In the bad old-days, MacOS X's HFS+ was not normalization-preserving. So it would force filenames to NFD form --- so if the user tried to create a file named Å, and passed in the Unicode string U+212B to creat(2), HFS+ would store it as U+0041,U+030A and that is what readdir(2) would return. Apple has effectively admitted this was a mistake, and their new APFS doesn't do this any more.

Now, both file systems basically say, "we don't care whether you pass in U+212B or U+0041,U+030A; on the screen it looks identical, Å, so we will treat it as the same filename; but readdir(2) will return what you gave us."

It's been a *long* time since Unicode has changed case folding rules for pre-existing characters. The tables have only changed with respect to the new character sets have been added. If you have a set of filenames which were all legal under Unicode 5.0, how they case fold didn't change with respect to Unicode 6.0, 7.0, 8.0 9.0, 10.0 or 11.0.

Unicode 11.0 added some character sets like Ancient Sanskrit, a bunch of new emoji's, and the copyleft symbol, and to the extent that Ancient Sanskrit had case, the tables might have been *extended*. But that doesn't break backwards compatibility.

And, of course, MacOS and Windows have been aggressively tracking Unicode updates because everybody wants the latest emoji's. :-)

And it's not just SAMBA/CIFS. The NFSv4 protocol also provides for case/normalization preserving filenames, and you can specify a NFSv4 mount option whether or not file name lookups should be case/normalization insensitive. And the NFSv4 protocol specs also specify the use of the Unicode thables, of which the latest versions can be downloaded here:

	http://www.unicode.org/Public/11.0.0/ucd/

So how about this? We'll put the unicode handling functions in a new directory, fs/unicode, just to make it really clear that this will now be changing any of the legacy fs/nls functions which other file systems will use. By putting it in a separate directory, it will be easier for other file systems to use it, whether it's for better Samba or NFSv4 support.

- Ted

#35 lore raw
Quoted 7 lines · Theodore Y. Ts'o

So things are much better in recent years. In the past it was kind of a disaster, but the world is converging enough that the latest versions of Mac OS'x APFS and Windows NTFS behave pretty much the same way. They are both case-insensitive, case-preserving and normalization-preserving, normalization-insensitive with respect to filenames.

Oh, so APFS at least fixed *that* horrific problem with their filesystem. Oh how I despised the exposure of NFD (which should at most be used as an internal representation, not externally visible). Turning basic letters (coming from Finland, åäö) into character combinations was an absolute abomination.

Quoted 1 line

In the bad old-days, MacOS X's HFS+ was not normalization-preserving.

Oh, I'm very aware.

It's not even that it wasn't normalization-preserving, it picked the *wrong* normalization to use.

Quoted 4 lines

Now, both file systems basically say, "we don't care whether you pass in U+212B or U+0041,U+030A; on the screen it looks identical, Å, so we will treat it as the same filename; but readdir(2) will return what you gave us."

Actually, the "on the screen it will look identical" is a horribly incorrect thing to do too.

There are lots of things that look identical on the screen without being at all the same thing. Sometimes it depends on font, sometimes it's just how it is. A nonbreaking space is *not* the same as a regular space, even if they may look identical on the screen.

I suspect (and sincerely _hope_) neither filesystem actually does anything as stuipid as taking "glyph equivalence" into account.

I'm hoping it's just "convert to NFx, then lower-case, then compare for equality". Where the 'x' doesn't much matter as long as it is never _exposed_ in any way outside of the comparison (ie NFD is a fine and probably simpler model for the lower-casing, the HFS+ mistake was to then expose the corrupted form of the filename).

Quoted 3 lines

It's been a *long* time since Unicode has changed case folding rules for pre-existing characters. The tables have only changed with respect to the new character sets have been added.

But new characters _have_ been added, and some of them do have lower-case form, so the folding tables have changed.

Happily, maybe that is over. As long as the Unicode people continue to mainly play with their Emoji list, I guess we can consider it done.

Quoted 6 lines

So how about this? We'll put the unicode handling functions in a new directory, fs/unicode, just to make it really clear that this will now be changing any of the legacy fs/nls functions which other file systems will use. By putting it in a separate directory, it will be easier for other file systems to use it, whether it's for better Samba or NFSv4 support.

Ok, that sounds fine.

Some of the unicode translation functions from the NLS code could well move into that, and NLS itself could be relegated to the sad historical thing.

And please try to make the *interfaces* sane.

For example, the interface for "let's compare with folded case" should *not* be about "convert to NFDK and lower case into a temp buffer, then compare the results".

You can do a lot of "let's handle the simple cases" faster even if the "oh, I hit a complex character" case might then become one of those "convert to a temp buffer" cases.

And it shouldn't be about C strings, since we very much have cases where it's not a C string but a {ptr,len} tuple. Maybe even use the "struct qstr", which is a not-horrible way to pass those around.

Even if you have a C string, you can always just do

        struct qstr str = QSTR_INIT(name, strlen(name));

and then pass that qstr pointer around.

Finally, don't do the NLS thing with "descriptors". that you register and look up. The indirection kills you. Particularly the crazy "one character at a time" model.

Just let people explicitly say "utf8_icasecmp(qstr, qstr)" or something like that. With the interface at least allowing for the common simple cases (ie everything is in the ASCII subset) to be handled basically as a specialized thing.

Linus

#36 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

On Sat, Dec 8, 2018 at 9:03 PM Theodore Y. Ts'o <[email protected]> wrote:

Either you have a horrible fundamental design mistake that has different per-filesystem locales, or you don't.

If you don't, you shouldn't be touching any of the nls code.

Whatever unicode tables you use for case folding shouldn't be in the nls code.

Hi Linus,

As Ted mentioned the SMB case, in my understanding, we might have more users for in-kernel ut8 normalization/casefold comparison functions than just ext4 in the future. Steve French (in cc.), for instance, mentioned his interest in using this higher level NLS API when I first submitted these patches.

My first RFC actually included this code as a separated module inside lib/ instead of touching NLS, but I found myself rewriting much of the same APIs that already existed in NLS. That is why I merged my work with that subsystem. I am open to rethinking it, if there is a better alternative.

Thanks,

Signature · 1 line
Gabriel Krisman Bertazi
#37 lore raw
Quoted 4 lines · Gabriel Krisman Bertazi

As Ted mentioned the SMB case, in my understanding, we might have more users for in-kernel ut8 normalization/casefold comparison functions than just ext4 in the future.

Crossed emails.

See my note about how there really is not a single case-folding library. It's simply not physically possible, because there are so many different ideas about what case-folding actually means.

That's still true even if "everything is utf-8", sadly.

So how do you handle locale issues and things like "we have ten different tables for utf-8 comparisons, and that's _ignoring_ the issue of whether we combine or decompose characters"?

And there's no way you can use the existing nls interfaces for upper/lower case, for example, since they are all limited to 256-byte tables and direct accesses to said tables, afaik.

And if that is where the extensions were, and that is why you changed other filesystems, this all matters.

My *guess* is that what you really want is not really about unicode at all, but specifically about just the NTFS rules. Which, yes, might find generic sharing interest between cifs/ext4/etc, but my gut feel is that they'd be specifically about some NTFS interoperability library.

Because even then I think you might have issues like "NTFS-5.1" vs "NTFS-4.0" etc.

Maybe you don't care, and you're picking just *one* version. And I haven't seen the code.

Basically, I would not be surprised if the sanest model is simply to make a "ntfs" library. Because I'm really fairly sure that OS X rules are very different indeed, even if it too is "unicode".

Linus