Skip to contentSkip to conversation

case-insensitive feature updates

PATCH Applied Gabriel Krisman Bertazi 3 participants 22 messages Apr 13, 2019 – May 2, 2019 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover case-insensitive feature updates

Hi,

The following patches fix a few bugs on the case-insensitive implementation on e2fsprogs, update to unicode 12.0 and implement the design decision of not normalizing file names on directories without the Casefold attribute.

Gabriel Krisman Bertazi (10):
e2p: encoding: Fix default flags lookup
ext2fs: Always attempt to load nls table when loading the filesystem
ext2fs: nls: Convert NFKD -> NFD
ext2fs: nls: Fix potential null dereference in utf8nlookup
ext2fs: nls: Merge sparse fixes from the kernel version
ext2fs: nls: Update to unicode 12.0.0
ext2fs: Don't normalize names on -F directories
ext2fs: nls: Drop normalization code
ext2fs: nls: Drop ascii NLS support
ext4.5.in: Document design changes on the casefold attribute

 e2fsck/unix.c              |   12 +-
 lib/e2p/encoding.c         |   12 +-
 lib/ext2fs/Makefile.in     |    5 +-
 lib/ext2fs/dirhash.c       |    8 +-
 lib/ext2fs/ext2_fs.h       |    5 +-
 lib/ext2fs/initialize.c    |    4 +
 lib/ext2fs/nls.h           |   10 +-
 lib/ext2fs/nls_ascii.c     |   68 -
 lib/ext2fs/nls_utf8-norm.c |   26 +-
 lib/ext2fs/nls_utf8.c      |   37 +-
 lib/ext2fs/openfs.c        |    4 +
 lib/ext2fs/utf8data.h      | 8889 ++++++++++++++----------------------
 lib/ext2fs/utf8n.h         |   14 +-
 misc/ext4.5.in             |   10 +-
 14 files changed, 3508 insertions(+), 5596 deletions(-)
 delete mode 100644 lib/ext2fs/nls_ascii.c
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/10 e2p: encoding: Fix default flags lookup

---
 lib/e2p/encoding.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/e2p/encoding.c +1 -1
index 88433310a1dd..27e93305172b 100644
@@ -67,7 +67,7 @@ int e2p_get_encoding_flags(int encoding)
 
 	for (i = 0 ; i < ARRAY_SIZE(ext4_encoding_map); i++)
 		if (ext4_encoding_map[i].encoding_magic == encoding)
-			return ext4_encoding_map[encoding].default_flags;
+			return ext4_encoding_map[i].default_flags;
 
 	return 0;
 }
Signature · 1 line
2.20.1
#3 lore raw

PATCH 2/10 ext2fs: Always attempt to load nls table when loading the filesystem

In reply to Gabriel Krisman Bertazi

fs->encoding is exposed by the library, so we need to at least try to load it when populating ext2_filsys. Nevertheless, failing to do so shouldn't be a fatal error, unless the user really needs that information. Thus, we ignore this failure during open/initialization and let the user who needs it validate that field before trying to use it.

---
 e2fsck/unix.c           | 12 ++++--------
 lib/ext2fs/initialize.c |  4 ++++
 lib/ext2fs/openfs.c     |  4 ++++
 3 files changed, 12 insertions(+), 8 deletions(-)
e2fsck/unix.c +4 -8
index 5b3552ece6b1..7c24f2e31e4f 100644
@@ -55,7 +55,6 @@ extern int optind;
 #include "problem.h"
 #include "jfs_user.h"
 #include "../version.h"
-#include <ext2fs/nls.h>
 
 /* Command line options */
 static int cflag;		/* check disk */
@@ -1785,13 +1784,10 @@ print_unsupp_features:
 		goto get_newer;
 	}
 
-	if (ext2fs_has_feature_fname_encoding(sb)) {
-		fs->encoding = nls_load_table(sb->s_encoding);
-		if (!fs->encoding) {
-			log_err(ctx, _("%s has unsupported encoding: %0x\n"),
-				ctx->filesystem_name, sb->s_encoding);
-			goto get_newer;
-		}
+	if (ext2fs_has_feature_fname_encoding(sb) && !fs->encoding) {
+		log_err(ctx, _("%s has unsupported encoding: %0x\n"),
+			ctx->filesystem_name, sb->s_encoding);
+		goto get_newer;
 	}
 
 	/*
lib/ext2fs/initialize.c +4 -0
index 30b1ae033340..2d470a070bcc 100644
@@ -27,6 +27,7 @@
 
 #include "ext2_fs.h"
 #include "ext2fs.h"
+#include "nls.h"
 
 #ifndef O_BINARY
 #define O_BINARY 0
@@ -190,6 +191,9 @@ errcode_t ext2fs_initialize(const char *name, int flags,
 	assign_field(s_encoding);
 	assign_field(s_encoding_flags);
 
+	if (ext2fs_has_feature_fname_encoding(param))
+		fs->encoding = nls_load_table(param->s_encoding);
+
 	if (super->s_feature_incompat & ~EXT2_LIB_FEATURE_INCOMPAT_SUPP) {
 		retval = EXT2_ET_UNSUPP_FEATURE;
 		goto cleanup;
lib/ext2fs/openfs.c +4 -0
index 85d73e2a429b..9ee6cd07f7f3 100644
@@ -32,6 +32,7 @@
 
 #include "ext2fs.h"
 #include "e2image.h"
+#include "nls.h"
 
 blk64_t ext2fs_descriptor_block_loc2(ext2_filsys fs, blk64_t group_block,
 				     dgrp_t i)
@@ -502,6 +503,9 @@ errcode_t ext2fs_open2(const char *name, const char *io_options,
 		ext2fs_set_feature_shared_blocks(fs->super);
 	}
 
+	if (ext2fs_has_feature_fname_encoding(fs->super))
+		fs->encoding = nls_load_table(fs->super->s_encoding);
+
 	fs->flags &= ~EXT2_FLAG_NOFREE_ON_ERROR;
 	*ret_fs = fs;
 
Signature · 1 line
2.20.1
#4 lore raw

Re: [PATCH e2fsprogs 02/10] ext2fs: Always attempt to load nls table when loading the filesystem

Quoted 8 lines · Gabriel Krisman Bertazi

fs->encoding is exposed by the library, so we need to at least try to load it when populating ext2_filsys. Nevertheless, failing to do so shouldn't be a fatal error, unless the user really needs that information. Thus, we ignore this failure during open/initialization and let the user who needs it validate that field before trying to use it.

Thanks, applied.

- Ted

#5 lore raw

PATCH 4/10 ext2fs: nls: Fix potential null dereference in utf8nlookup

In reply to Gabriel Krisman Bertazi

Delay the access of data->offset until after the null check. This was reported by 0-day on the kernel version of the script.

---
 lib/ext2fs/nls_utf8-norm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
lib/ext2fs/nls_utf8-norm.c +3 -1
index 5a91573f766f..4230b64bc586 100644
@@ -335,7 +335,7 @@ utf8hangul(const char *str, unsigned char *hangul)
 static utf8leaf_t *utf8nlookup(const struct utf8data *data,
 			       unsigned char *hangul, const char *s, size_t len)
 {
-	utf8trie_t	*trie = utf8data + data->offset;
+	utf8trie_t	*trie = NULL;
 	int		offlen;
 	int		offset;
 	int		mask;
@@ -345,6 +345,8 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data,
 		return NULL;
 	if (len == 0)
 		return NULL;
+
+	trie = utf8data + data->offset;
 	node = 1;
 	while (node) {
 		offlen = (*trie & OFFLEN) >> OFFLEN_SHIFT;
Signature · 1 line
2.20.1
#6 lore raw

Re: [PATCH e2fsprogs 04/10] ext2fs: nls: Fix potential null dereference in utf8nlookup

Quoted 4 lines · Gabriel Krisman Bertazi

Delay the access of data->offset until after the null check. This was reported by 0-day on the kernel version of the script.

Applied, thanks.

- Ted

#7 lore raw

PATCH 5/10 ext2fs: nls: Merge sparse fixes from the kernel version

In reply to Gabriel Krisman Bertazi

The sparse script complained about the following issues in the kernel version of this script. This patch port the fixes to the userspace version.

sparse warnings: (new ones prefixed by >>)

Quoted 1 line
Quoted 1 line

fs/unicode/utf8-norm.c:41:24: sparse: non-ANSI function declaration

of function 'utf8version_latest'

vim +/utf8version_latest +41 fs/unicode/utf8-norm.c

40

Quoted 1 line

41 int utf8version_latest()

42 {
43 return utf8vers;
44 }
45 EXPORT_SYMBOL(utf8version_latest);
46

---
 lib/ext2fs/nls_utf8-norm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/ext2fs/nls_utf8-norm.c +1 -1
index 4230b64bc586..27e530c14e4f 100644
@@ -46,7 +46,7 @@ int utf8version_is_supported(uint8_t maj, uint8_t min, uint8_t rev)
 	return 0;
 }
 
-int utf8version_latest()
+int utf8version_latest(void)
 {
 	return utf8vers;
 }
Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH e2fsprogs 05/10] ext2fs: nls: Merge sparse fixes from the kernel version

Quoted 12 lines · Gabriel Krisman Bertazi

The sparse script complained about the following issues in the kernel version of this script. This patch port the fixes to the userspace version.

sparse warnings: (new ones prefixed by >>)

Quoted 1 line
Quoted 1 line

fs/unicode/utf8-norm.c:41:24: sparse: non-ANSI function declaration

of function 'utf8version_latest'

vim +/utf8version_latest +41 fs/unicode/utf8-norm.c

40

Quoted 8 lines
Quoted 1 line

41 int utf8version_latest()

42 {
43 return utf8vers;
44 }
45 EXPORT_SYMBOL(utf8version_latest);
46

Thanks, applied.

- Ted

#9 lore raw

PATCH 7/10 ext2fs: Don't normalize names on -F directories

In reply to Gabriel Krisman Bertazi

As agreed on the new design, even if fname_encoding is enabled, directories entries who aren't owned by a +F directory will not be normalized.

---
 lib/ext2fs/dirhash.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
lib/ext2fs/dirhash.c +2 -6
index f1e7734d3865..b9afb599c0fa 100644
@@ -281,15 +281,11 @@ errcode_t ext2fs_dirhash2(int version, const char *name, int len,
 	int dlen;
 	unsigned char *buff;
 
-	if (len && charset) {
+	if (len && charset && (hash_flags & EXT4_CASEFOLD_FL)) {
 		char buff[PATH_MAX];
 
-		if (hash_flags & EXT4_CASEFOLD_FL)
-			dlen = charset->ops->casefold(charset, name, len, buff,
+		dlen = charset->ops->casefold(charset, name, len, buff,
 						      sizeof(buff));
-		else
-			dlen = charset->ops->normalize(charset, name, len, buff,
-						       sizeof(buff));
 		if (dlen < 0) {
 			if (dlen == -EINVAL)
 				goto opaque_seq;
Signature · 1 line
2.20.1
#10 lore raw

Re: [PATCH e2fsprogs 07/10] ext2fs: Don't normalize names on -F directories

Quoted 5 lines · Gabriel Krisman Bertazi

As agreed on the new design, even if fname_encoding is enabled, directories entries who aren't owned by a +F directory will not be normalized.

Thanks, applied.

- Ted

#11 lore raw

PATCH 8/10 ext2fs: nls: Drop normalization code

In reply to Gabriel Krisman Bertazi

No longer necessary after +F design change.

---
 lib/e2p/encoding.c     |  3 +--
 lib/ext2fs/ext2_fs.h   |  2 --
 lib/ext2fs/nls.h       |  4 ----
 lib/ext2fs/nls_ascii.c | 20 --------------------
 lib/ext2fs/nls_utf8.c  | 31 -------------------------------
 5 files changed, 1 insertion(+), 59 deletions(-)
lib/e2p/encoding.c +1 -2
index 23470bdf38a5..62e0aada92c2 100644
@@ -36,8 +36,7 @@ static const struct {
 	{
 		.encoding_magic = EXT4_ENC_UTF8_12_0,
 		.name = "utf8",
-		.default_flags = (EXT4_UTF8_NORMALIZATION_TYPE_NFKD |
-				  EXT4_UTF8_CASEFOLD_TYPE_NFKDCF)
+		.default_flags = 0,
 	},
 };
 
lib/ext2fs/ext2_fs.h +0 -2
index a1edc54dfd52..80ff02265ee5 100644
@@ -1132,7 +1132,5 @@ struct mmp_struct {
 #define EXT4_ENC_UTF8_12_0	1
 
 #define EXT4_ENC_STRICT_MODE_FL			(1 << 0) /* Reject invalid sequences */
-#define EXT4_UTF8_NORMALIZATION_TYPE_NFKD	(1 << 1)
-#define EXT4_UTF8_CASEFOLD_TYPE_NFKDCF		(1 << 4)
 
 #endif	/* _LINUX_EXT2_FS_H */
lib/ext2fs/nls.h +0 -4
index c4fcbcb32528..ae56aa9abc89 100644
@@ -33,10 +33,6 @@ struct nls_table;
         (sizeof(array) / sizeof(array[0]))
 
 struct nls_ops {
-	int (*normalize)(const struct nls_table *charset,
-			 const unsigned char *str, size_t len,
-			 unsigned char *dest, size_t dlen);
-
 	int (*casefold)(const struct nls_table *charset,
 			const unsigned char *str, size_t len,
 			unsigned char *dest, size_t dlen);
lib/ext2fs/nls_ascii.c +0 -20
index 5d513df404c1..cc03fea97ec5 100644
@@ -39,28 +39,8 @@ static int ascii_casefold(const struct nls_table *table,
 	return len;
 }
 
-static int ascii_normalize(const struct nls_table *table,
-			   const unsigned char *str, size_t len,
-			   unsigned char *dest, size_t dlen)
-{
-	int i;
-
-	if (dlen < len)
-		return -ENAMETOOLONG;
-
-	for (i = 0; i < len; i++) {
-		if (str[i] & 0x80)
-			return -EINVAL;
-
-		dest[i] = str[i];
-	}
-
-	return len;
-}
-
 const static struct nls_ops ascii_ops = {
 	.casefold = ascii_casefold,
-	.normalize = ascii_normalize,
 };
 
 const struct nls_table nls_ascii = {
lib/ext2fs/nls_utf8.c +0 -31
index 426657764fc6..b1698e3c409f 100644
@@ -54,39 +54,8 @@ invalid_seq:
 	return -EINVAL;
 }
 
-static int utf8_normalize(const struct nls_table *table,
-			  const unsigned char *str, size_t len,
-			  unsigned char *dest, size_t dlen)
-{
-	const struct utf8data *data = utf8nfdi(table->version);
-	struct utf8cursor cur;
-	ssize_t nlen = 0;
-
-	if (utf8ncursor(&cur, data, str, len) < 0)
-		goto invalid_seq;
-
-	for (nlen = 0; nlen < dlen; nlen++) {
-		dest[nlen] = utf8byte(&cur);
-		if (!dest[nlen])
-			return nlen;
-		if (dest[nlen] == -1)
-			break;
-	}
-
-	return -ENAMETOOLONG;
-
-invalid_seq:
-	if (dlen < len)
-		return -ENAMETOOLONG;
-
-	/* Signal invalid sequence */
-	return -EINVAL;
-}
-
 const static struct nls_ops utf8_ops = {
 	.casefold = utf8_casefold,
-	.normalize = utf8_normalize,
-
 };
 
 const struct nls_table nls_utf8_12_0 = {
Signature · 1 line
2.20.1
#13 lore raw

Change feature name from fname_encoding to casefold?

In reply to Gabriel Krisman Bertazi

Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from fname_encoding to casefold.

What do you think? Any objections?

- Ted

#14 lore raw

Re: Change feature name from fname_encoding to casefold?

Quoted 6 lines · Theodore Ts'o

Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from fname_encoding to casefold.

What do you think? Any objections?

I don't object at all, but I'll need to update some test packages/documentation already published to customers in order to change it. So I'd need a final decision to be taken quickly on it. :)

Signature · 1 line
Gabriel Krisman Bertazi
#15 lore raw

Re: Change feature name from fname_encoding to casefold?

In reply to Theodore Ts'o
Quoted 4 lines · Theodore Ts'o

Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from fname_encoding to casefold.

TBH /me hadn't done enough reviewing even to notice the feature was named 'fname_encoding' (whatever that means -- encoded how?). IMHO 'casefold' is more descriptive about what the feature provides (folding case for directory name comparisons, right?)

--D

Quoted 4 lines

What do you think? Any objections?

- Ted

#16 lore raw

Re: Change feature name from fname_encoding to casefold?

Quoted 11 lines · Darrick J. Wong
Quoted 4 lines · Theodore Ts'o

Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from fname_encoding to casefold.

TBH /me hadn't done enough reviewing even to notice the feature was named 'fname_encoding' (whatever that means -- encoded how?). IMHO 'casefold' is more descriptive about what the feature provides (folding case for directory name comparisons, right?)

This name was accurate until the v6 of the patches. In v7, we decided to no longer apply file name encoding normalization system-wide during lookups, which changed the semantics of this feature flag. By that time, though, the name was already committed to e2fsprogs, so i didn't bother to change it.

Signature · 1 line
Gabriel Krisman Bertazi
#17 lore raw

PATCH 9/10 ext2fs: nls: Drop ascii NLS support

In reply to Gabriel Krisman Bertazi

As agreed on the kernel side, ext4 will only support utf8 for now, and not plain ascii, so we don't need this anymore.

---
 lib/e2p/encoding.c     |  5 -----
 lib/ext2fs/Makefile.in |  5 ++---
 lib/ext2fs/ext2_fs.h   |  1 -
 lib/ext2fs/nls.h       |  2 --
 lib/ext2fs/nls_ascii.c | 48 ------------------------------------------
 5 files changed, 2 insertions(+), 59 deletions(-)
 delete mode 100644 lib/ext2fs/nls_ascii.c
lib/e2p/encoding.c +0 -5
index 62e0aada92c2..6a3c472b1b31 100644
@@ -28,11 +28,6 @@ static const struct {
 	__u16 default_flags;
 
 } ext4_encoding_map[] = {
-	{
-		.encoding_magic = EXT4_ENC_ASCII,
-		.name = "ascii",
-		.default_flags = 0
-	},
 	{
 		.encoding_magic = EXT4_ENC_UTF8_12_0,
 		.name = "utf8",
lib/ext2fs/Makefile.in +2 -3
index 06c2394b34b8..4b3fc1881cff 100644
@@ -20,8 +20,8 @@ COMPILE_ET=	_ET_DIR_OVERRIDE=$(srcdir)/../et ../et/compile_et
 @TEST_IO_CMT@TEST_IO_LIB_OBJS = test_io.o
 @IMAGER_CMT@E2IMAGE_LIB_OBJS = imager.o
 
-NLS_OBJS=nls_ascii.o nls_utf8-norm.o nls_utf8.o
-NLS_SRCS=$(srcdir)/nls_ascii.c $(srcdir)/nls_utf8-norm.c $(srcdir)/nls_utf8.c
+NLS_OBJS=nls_utf8-norm.o nls_utf8.o
+NLS_SRCS=$(srcdir)/nls_utf8-norm.c $(srcdir)/nls_utf8.c
 
 DEBUG_OBJS= debug_cmds.o extent_cmds.o tst_cmds.o debugfs.o util.o \
 	ncheck.o icheck.o ls.o lsdel.o dump.o set_fields.o logdump.o \
@@ -1328,7 +1328,6 @@ do_journal.o: $(top_srcdir)/debugfs/do_journal.c $(top_builddir)/lib/config.h \
  $(top_srcdir)/lib/support/quotaio_tree.h $(srcdir)/kernel-jbd.h \
  $(srcdir)/jfs_compat.h $(srcdir)/kernel-list.h \
  $(top_srcdir)/debugfs/journal.h $(srcdir)/../../e2fsck/jfs_user.h
-nls_ascii.o: $(srcdir)/nls_ascii.c $(srcdir)/nls.h $(srcdir)/ext2_fs.h
 nls_utf8-norm.o: $(srcdir)/nls_utf8-norm.c $(srcdir)/utf8n.h \
  $(srcdir)/utf8data.h
 nls_utf8.o: $(srcdir)/nls_utf8.c $(srcdir)/nls.h $(srcdir)/ext2_fs.h \
lib/ext2fs/ext2_fs.h +0 -1
index 80ff02265ee5..fc8a650a3aad 100644
@@ -1128,7 +1128,6 @@ struct mmp_struct {
  */
 #define EXT4_INLINE_DATA_DOTDOT_SIZE	(4)
 
-#define EXT4_ENC_ASCII		0
 #define EXT4_ENC_UTF8_12_0	1
 
 #define EXT4_ENC_STRICT_MODE_FL			(1 << 0) /* Reject invalid sequences */
lib/ext2fs/nls.h +0 -2
index ae56aa9abc89..f008bd167f36 100644
@@ -43,14 +43,12 @@ struct nls_table {
 	const struct nls_ops *ops;
 };
 
-extern const struct nls_table nls_ascii;
 extern const struct nls_table nls_utf8_12_0;
 
 static const struct {
 	int encoding_magic;
 	const struct nls_table *tbl;
 } nls_map[] = {
-	{ EXT4_ENC_ASCII, &nls_ascii },
 	{ EXT4_ENC_UTF8_12_0, &nls_utf8_12_0 },
 };
 
lib/ext2fs/nls_ascii.c deleted +0 -48

48 lines not on this page · raw message

Signature · 1 line
2.20.1
#18 lore raw

Re: [PATCH e2fsprogs 09/10] ext2fs: nls: Drop ascii NLS support

Quoted 4 lines · Gabriel Krisman Bertazi

As agreed on the kernel side, ext4 will only support utf8 for now, and not plain ascii, so we don't need this anymore.

Thanks, applied.

- Ted

#19 lore raw

PATCH 10/10 ext4.5.in: Document design changes on the casefold attribute

In reply to Gabriel Krisman Bertazi

Document the fact that the encoding support is only used by directories with the +F attribute.

---
 misc/ext4.5.in | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
misc/ext4.5.in +4 -6
index ee5e170979a1..2a8ce1e2f8e8 100644
@@ -154,12 +154,10 @@ option.
 .B fname_encoding
 .br
 This ext4 feature provides file system level character encoding support
-for files and directories name.  This feature is name-preserving on the
-disk, but it allows applications to lookup for a file in the file system
-using any encoding equivalent version of the file name.
-
-This feature is required to perform in-kernel case-insensitive file
-name lookups.
+for directories with the casefold (+F) flag enabled.  This feature is
+name-preserving on the disk, but it allows applications to lookup for a
+file in the file system using an encoding equivalent version of the file
+name.
 .TP
 .B has_journal
 .br
Signature · 1 line
2.20.1
#20 lore raw

Re: [PATCH e2fsprogs 10/10] ext4.5.in: Document design changes on the casefold attribute

Quoted 4 lines · Gabriel Krisman Bertazi

Document the fact that the encoding support is only used by directories with the +F attribute.

Thanks, applied.

- Ted

A message that is not in the archive (sent off-list or not yet received): [email protected]

#22 lore raw

Re: [PATCH e2fsprogs 03/10] ext2fs: nls: Convert NFKD -> NFD

Quoted 4 lines · Gabriel Krisman Bertazi

Following the kernel transition, convert the normalization format from NFKD to NFD. This also regenerates the utf8data.h database.

Thanks, applied.

- Ted