case-insensitive feature updates
cover case-insensitive feature updates
Hi,
The following patches fix a few bugs on the case-insensitive implementation on e2fsprogs, update to unicode 12.0 and implement the design decision of not normalizing file names on directories without the Casefold attribute.
Gabriel Krisman Bertazi (10):
e2p: encoding: Fix default flags lookup
ext2fs: Always attempt to load nls table when loading the filesystem
ext2fs: nls: Convert NFKD -> NFD
ext2fs: nls: Fix potential null dereference in utf8nlookup
ext2fs: nls: Merge sparse fixes from the kernel version
ext2fs: nls: Update to unicode 12.0.0
ext2fs: Don't normalize names on -F directories
ext2fs: nls: Drop normalization code
ext2fs: nls: Drop ascii NLS support
ext4.5.in: Document design changes on the casefold attribute
e2fsck/unix.c | 12 +- lib/e2p/encoding.c | 12 +- lib/ext2fs/Makefile.in | 5 +- lib/ext2fs/dirhash.c | 8 +- lib/ext2fs/ext2_fs.h | 5 +- lib/ext2fs/initialize.c | 4 + lib/ext2fs/nls.h | 10 +- lib/ext2fs/nls_ascii.c | 68 - lib/ext2fs/nls_utf8-norm.c | 26 +- lib/ext2fs/nls_utf8.c | 37 +- lib/ext2fs/openfs.c | 4 + lib/ext2fs/utf8data.h | 8889 ++++++++++++++---------------------- lib/ext2fs/utf8n.h | 14 +- misc/ext4.5.in | 10 +- 14 files changed, 3508 insertions(+), 5596 deletions(-) delete mode 100644 lib/ext2fs/nls_ascii.c
Signature · 1 line
2.20.1
PATCH 1/10 e2p: encoding: Fix default flags lookup
--- lib/e2p/encoding.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/e2p/encoding.c +1 -1
@@ -67,7 +67,7 @@ int e2p_get_encoding_flags(int encoding)for (i =0 ; i <ARRAY_SIZE (ext4_encoding_map); i++)if (ext4_encoding_map[i].encoding_magic == encoding) -return ext4_encoding_map[encoding].default_flags; +return ext4_encoding_map[i].default_flags;return 0 ; }
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 01/10] e2p: encoding: Fix default flags lookup
Quoted 1 line · Gabriel Krisman Bertazi
Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
PATCH 2/10 ext2fs: Always attempt to load nls table when loading the filesystem
In reply to Gabriel Krisman Bertazifs->encoding is exposed by the library, so we need to at least try to load it when populating ext2_filsys. Nevertheless, failing to do so shouldn't be a fatal error, unless the user really needs that information. Thus, we ignore this failure during open/initialization and let the user who needs it validate that field before trying to use it.
--- e2fsck/unix.c | 12 ++++-------- lib/ext2fs/initialize.c | 4 ++++ lib/ext2fs/openfs.c | 4 ++++ 3 files changed, 12 insertions(+), 8 deletions(-)
e2fsck/unix.c +4 -8
@@ -55,7 +55,6 @@ extern int optind;#include "problem.h" #include "jfs_user.h" #include "../version.h" -#include <ext2fs/nls.h> /* Command line options */ static int cflag;/* check disk */
@@ -1785,13 +1784,10 @@ print_unsupp_features: goto get_newer; } -if (ext2fs_has_feature_fname_encoding (sb)) { - fs->encoding =nls_load_table (sb->s_encoding); -if (!fs->encoding) { -log_err (ctx,_ ("%s has unsupported encoding: %0x\n" ), - ctx->filesystem_name, sb->s_encoding); - goto get_newer; - } +if (ext2fs_has_feature_fname_encoding (sb) && !fs->encoding) { +log_err (ctx,_ ("%s has unsupported encoding: %0x\n" ), + ctx->filesystem_name, sb->s_encoding); + goto get_newer; }/*
lib/ext2fs/initialize.c +4 -0
@@ -27,6 +27,7 @@#include "ext2_fs.h" #include "ext2fs.h" +#include "nls.h" #ifndef O_BINARY #define O_BINARY 0
@@ -190,6 +191,9 @@ errcode_t ext2fs_initialize(const char *name, int flags,assign_field (s_encoding);assign_field (s_encoding_flags); +if (ext2fs_has_feature_fname_encoding (param)) + fs->encoding =nls_load_table (param->s_encoding); +if (super->s_feature_incompat & ~EXT2_LIB_FEATURE_INCOMPAT_SUPP ) { retval =EXT2_ET_UNSUPP_FEATURE ; goto cleanup;
lib/ext2fs/openfs.c +4 -0
@@ -32,6 +32,7 @@#include "ext2fs.h" #include "e2image.h" +#include "nls.h" blk64_t ext2fs_descriptor_block_loc2 (ext2_filsys fs,blk64_t group_block,dgrp_t i)
@@ -502,6 +503,9 @@ errcode_t ext2fs_open2(const char *name, const char *io_options,ext2fs_set_feature_shared_blocks (fs->super); } +if (ext2fs_has_feature_fname_encoding (fs->super)) + fs->encoding =nls_load_table (fs->super->s_encoding); + fs->flags &= ~EXT2_FLAG_NOFREE_ON_ERROR ; *ret_fs = fs;
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 02/10] ext2fs: Always attempt to load nls table when loading the filesystem
Quoted 8 lines · Gabriel Krisman Bertazi
fs->encodingis exposed by the library, so we need to at least try to load it when populatingext2_filsys. Nevertheless, failing to do so shouldn't be a fatal error, unless the user really needs that information. Thus, we ignore this failure during open/initialization and let the user who needs it validate that field before trying to use it.Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
PATCH 4/10 ext2fs: nls: Fix potential null dereference in utf8nlookup
In reply to Gabriel Krisman BertaziDelay the access of data->offset until after the null check. This was reported by 0-day on the kernel version of the script.
--- lib/ext2fs/nls_utf8-norm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
lib/ext2fs/nls_utf8-norm.c +3 -1
@@ -335,7 +335,7 @@ utf8hangul(const char *str, unsigned char *hangul)static utf8leaf_t *utf8nlookup (const struct utf8data *data,unsigned char *hangul,const char *s,size_t len) { -utf8trie_t *trie = utf8data + data->offset; +utf8trie_t *trie =NULL ;int offlen;int offset;int mask;
@@ -345,6 +345,8 @@ static utf8leaf_t *utf8nlookup(const struct utf8data *data,return NULL ;if (len ==0 )return NULL ; + + trie = utf8data + data->offset; node =1 ;while (node) { offlen = (*trie &OFFLEN ) >>OFFLEN_SHIFT ;
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 04/10] ext2fs: nls: Fix potential null dereference in utf8nlookup
Quoted 4 lines · Gabriel Krisman Bertazi
Delay the access of
data->offsetuntil after the null check. This was reported by 0-day on the kernel version of the script.Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Applied, thanks.
- Ted
PATCH 5/10 ext2fs: nls: Merge sparse fixes from the kernel version
In reply to Gabriel Krisman BertaziThe sparse script complained about the following issues in the kernel version of this script. This patch port the fixes to the userspace version.
sparse warnings: (new ones prefixed by >>)
Quoted 1 line
Quoted 1 line
fs/unicode/utf8-norm.c:41:24: sparse: non-ANSI function declaration
of function 'utf8version_latest'
vim +/utf8version_latest +41 fs/unicode/utf8-norm.c
40
Quoted 1 line
41 int
utf8version_latest()
42 {
43 return utf8vers;
44 }
45 EXPORT_SYMBOL(utf8version_latest);
46
--- lib/ext2fs/nls_utf8-norm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/ext2fs/nls_utf8-norm.c +1 -1
@@ -46,7 +46,7 @@ int utf8version_is_supported(uint8_t maj, uint8_t min, uint8_t rev)return 0 ; } -int utf8version_latest () +int utf8version_latest (void ) {return utf8vers; }
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 05/10] ext2fs: nls: Merge sparse fixes from the kernel version
Quoted 12 lines · Gabriel Krisman Bertazi
The sparse script complained about the following issues in the kernel version of this script. This patch port the fixes to the userspace version.
sparse warnings: (new ones prefixed by >>)
Quoted 1 line
Quoted 1 line
fs/unicode/utf8-norm.c:41:24: sparse: non-ANSI function declarationof function '
utf8version_latest'vim +/utf8version_latest +41
fs/unicode/utf8-norm.c40
Quoted 8 lines
Quoted 1 line
41 int
utf8version_latest()42 {
43 return utf8vers;
44 }
45EXPORT_SYMBOL(utf8version_latest);
46Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
PATCH 7/10 ext2fs: Don't normalize names on -F directories
In reply to Gabriel Krisman BertaziAs agreed on the new design, even if fname_encoding is enabled, directories entries who aren't owned by a +F directory will not be normalized.
--- lib/ext2fs/dirhash.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
lib/ext2fs/dirhash.c +2 -6
@@ -281,15 +281,11 @@ errcode_t ext2fs_dirhash2(int version, const char *name, int len,int dlen;unsigned char *buff; -if (len && charset) { +if (len && charset && (hash_flags &EXT4_CASEFOLD_FL )) {char buff[PATH_MAX ]; -if (hash_flags &EXT4_CASEFOLD_FL ) - dlen = charset->ops->casefold (charset, name, len, buff, + dlen = charset->ops->casefold (charset, name, len, buff,sizeof (buff)); -else - dlen = charset->ops->normalize (charset, name, len, buff, -sizeof (buff));if (dlen <0 ) {if (dlen == -EINVAL ) goto opaque_seq;
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 07/10] ext2fs: Don't normalize names on -F directories
Quoted 5 lines · Gabriel Krisman Bertazi
As agreed on the new design, even if
fname_encodingis enabled, directories entries who aren't owned by a +F directory will not be normalized.Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
PATCH 8/10 ext2fs: nls: Drop normalization code
In reply to Gabriel Krisman BertaziNo longer necessary after +F design change.
--- lib/e2p/encoding.c | 3 +-- lib/ext2fs/ext2_fs.h | 2 -- lib/ext2fs/nls.h | 4 ---- lib/ext2fs/nls_ascii.c | 20 -------------------- lib/ext2fs/nls_utf8.c | 31 ------------------------------- 5 files changed, 1 insertion(+), 59 deletions(-)
lib/e2p/encoding.c +1 -2
@@ -36,8 +36,7 @@ static const struct { { .encoding_magic = EXT4_ENC_UTF8_12_0, .name = "utf8", - .default_flags = (EXT4_UTF8_NORMALIZATION_TYPE_NFKD | - EXT4_UTF8_CASEFOLD_TYPE_NFKDCF) + .default_flags = 0, }, };
lib/ext2fs/ext2_fs.h +0 -2
@@ -1132,7 +1132,5 @@ struct mmp_struct {#define EXT4_ENC_UTF8_12_0 1#define EXT4_ENC_STRICT_MODE_FL (1 << 0)/* Reject invalid sequences */ -#define EXT4_UTF8_NORMALIZATION_TYPE_NFKD (1 << 1) -#define EXT4_UTF8_CASEFOLD_TYPE_NFKDCF (1 << 4)#endif /* _LINUX_EXT2_FS_H */
lib/ext2fs/nls.h +0 -4
@@ -33,10 +33,6 @@ struct nls_table; (sizeof(array) / sizeof(array[0]))struct nls_ops { -int (*normalize)(const struct nls_table *charset, -const unsigned char *str,size_t len, -unsigned char *dest,size_t dlen); -int (*casefold)(const struct nls_table *charset,const unsigned char *str,size_t len,unsigned char *dest,size_t dlen);
lib/ext2fs/nls_ascii.c +0 -20
@@ -39,28 +39,8 @@ static int ascii_casefold(const struct nls_table *table,return len; } -static int ascii_normalize (const struct nls_table *table, -const unsigned char *str,size_t len, -unsigned char *dest,size_t dlen) -{ -int i; - -if (dlen < len) -return -ENAMETOOLONG ; - -for (i =0 ; i < len; i++) { -if (str[i] &0x80 ) -return -EINVAL ; - - dest[i] = str[i]; - } - -return len; -} -const static struct nls_ops ascii_ops = { .casefold = ascii_casefold, - .normalize = ascii_normalize, };const struct nls_table nls_ascii = {
lib/ext2fs/nls_utf8.c +0 -31
@@ -54,39 +54,8 @@ invalid_seq:return -EINVAL ; } -static int utf8_normalize (const struct nls_table *table, -const unsigned char *str,size_t len, -unsigned char *dest,size_t dlen) -{ -const struct utf8data *data =utf8nfdi (table->version); -struct utf8cursor cur; -ssize_t nlen =0 ; - -if (utf8ncursor (&cur, data, str, len) <0 ) - goto invalid_seq; - -for (nlen =0 ; nlen < dlen; nlen++) { - dest[nlen] =utf8byte (&cur); -if (!dest[nlen]) -return nlen; -if (dest[nlen] ==-1 ) -break ; - } - -return -ENAMETOOLONG ; - -invalid_seq: -if (dlen < len) -return -ENAMETOOLONG ; - -/* Signal invalid sequence */ -return -EINVAL ; -} -const static struct nls_ops utf8_ops = { .casefold = utf8_casefold, - .normalize = utf8_normalize, - };const struct nls_table nls_utf8_12_0 = {
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 08/10] ext2fs: nls: Drop normalization code
Quoted 3 lines · Gabriel Krisman Bertazi
No longer necessary after +F design change.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
Change feature name from fname_encoding to casefold?
In reply to Gabriel Krisman BertaziGiven how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from fname_encoding to casefold.
What do you think? Any objections?
- Ted
Re: Change feature name from fname_encoding to casefold?
Quoted 6 lines · Theodore Ts'o
Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from
fname_encodingto casefold.What do you think? Any objections?
I don't object at all, but I'll need to update some test packages/documentation already published to customers in order to change it. So I'd need a final decision to be taken quickly on it. :)
Signature · 1 line
Gabriel Krisman Bertazi
Re: Change feature name from fname_encoding to casefold?
In reply to Theodore Ts'oQuoted 4 lines · Theodore Ts'o
Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from
fname_encodingto casefold.
TBH /me hadn't done enough reviewing even to notice the feature was named 'fname_encoding' (whatever that means -- encoded how?). IMHO 'casefold' is more descriptive about what the feature provides (folding case for directory name comparisons, right?)
--D
Quoted 4 lines
What do you think? Any objections?
- Ted
Re: Change feature name from fname_encoding to casefold?
Quoted 11 lines · Darrick J. Wong
Quoted 4 lines · Theodore Ts'o
Given how we've simplified how we handle Unicode --- in particular, not doing any kind of normalization unless we are doing case-folding compares, I think it will be more user-friendly if we rename the feature from
fname_encodingto casefold.TBH /me hadn't done enough reviewing even to notice the feature was named '
fname_encoding' (whatever that means -- encoded how?). IMHO 'casefold' is more descriptive about what the feature provides (folding case for directory name comparisons, right?)
This name was accurate until the v6 of the patches. In v7, we decided to no longer apply file name encoding normalization system-wide during lookups, which changed the semantics of this feature flag. By that time, though, the name was already committed to e2fsprogs, so i didn't bother to change it.
Signature · 1 line
Gabriel Krisman Bertazi
PATCH 9/10 ext2fs: nls: Drop ascii NLS support
In reply to Gabriel Krisman BertaziAs agreed on the kernel side, ext4 will only support utf8 for now, and not plain ascii, so we don't need this anymore.
--- lib/e2p/encoding.c | 5 ----- lib/ext2fs/Makefile.in | 5 ++--- lib/ext2fs/ext2_fs.h | 1 - lib/ext2fs/nls.h | 2 -- lib/ext2fs/nls_ascii.c | 48 ------------------------------------------ 5 files changed, 2 insertions(+), 59 deletions(-) delete mode 100644 lib/ext2fs/nls_ascii.c
lib/e2p/encoding.c +0 -5
@@ -28,11 +28,6 @@ static const struct {__u16 default_flags; } ext4_encoding_map[] = { - { - .encoding_magic = EXT4_ENC_ASCII, - .name = "ascii", - .default_flags =0 - }, { .encoding_magic = EXT4_ENC_UTF8_12_0, .name = "utf8",
lib/ext2fs/Makefile.in +2 -3
@@ -20,8 +20,8 @@ COMPILE_ET= _ET_DIR_OVERRIDE=$(srcdir)/../et ../et/compile_et @TEST_IO_CMT@TEST_IO_LIB_OBJS = test_io.o @IMAGER_CMT@E2IMAGE_LIB_OBJS = imager.o -NLS_OBJS=nls_ascii.o nls_utf8-norm.o nls_utf8.o -NLS_SRCS=$(srcdir)/nls_ascii.c $(srcdir)/nls_utf8-norm.c $(srcdir)/nls_utf8.c +NLS_OBJS=nls_utf8-norm.o nls_utf8.o +NLS_SRCS=$(srcdir)/nls_utf8-norm.c $(srcdir)/nls_utf8.c DEBUG_OBJS= debug_cmds.o extent_cmds.o tst_cmds.o debugfs.o util.o \ ncheck.o icheck.o ls.o lsdel.o dump.o set_fields.o logdump.o \
@@ -1328,7 +1328,6 @@ do_journal.o: $(top_srcdir)/debugfs/do_journal.c $(top_builddir)/lib/config.h \ $(top_srcdir)/lib/support/quotaio_tree.h $(srcdir)/kernel-jbd.h \ $(srcdir)/jfs_compat.h $(srcdir)/kernel-list.h \ $(top_srcdir)/debugfs/journal.h $(srcdir)/../../e2fsck/jfs_user.h -nls_ascii.o: $(srcdir)/nls_ascii.c $(srcdir)/nls.h $(srcdir)/ext2_fs.h nls_utf8-norm.o: $(srcdir)/nls_utf8-norm.c $(srcdir)/utf8n.h \ $(srcdir)/utf8data.h nls_utf8.o: $(srcdir)/nls_utf8.c $(srcdir)/nls.h $(srcdir)/ext2_fs.h \
lib/ext2fs/ext2_fs.h +0 -1
@@ -1128,7 +1128,6 @@ struct mmp_struct {*/ #define EXT4_INLINE_DATA_DOTDOT_SIZE (4) -#define EXT4_ENC_ASCII 0#define EXT4_ENC_UTF8_12_0 1#define EXT4_ENC_STRICT_MODE_FL (1 << 0)/* Reject invalid sequences */
lib/ext2fs/nls.h +0 -2
@@ -43,14 +43,12 @@ struct nls_table {const struct nls_ops *ops; }; -extern const struct nls_table nls_ascii;extern const struct nls_table nls_utf8_12_0;static const struct {int encoding_magic;const struct nls_table *tbl; } nls_map[] = { - {EXT4_ENC_ASCII , &nls_ascii }, {EXT4_ENC_UTF8_12_0 , &nls_utf8_12_0 }, };
lib/ext2fs/nls_ascii.c deleted +0 -48
48 lines not on this page · raw message
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 09/10] ext2fs: nls: Drop ascii NLS support
Quoted 4 lines · Gabriel Krisman Bertazi
As agreed on the kernel side, ext4 will only support utf8 for now, and not plain ascii, so we don't need this anymore.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
PATCH 10/10 ext4.5.in: Document design changes on the casefold attribute
In reply to Gabriel Krisman BertaziDocument the fact that the encoding support is only used by directories with the +F attribute.
--- misc/ext4.5.in | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
misc/ext4.5.in +4 -6
@@ -154,12 +154,10 @@ option. .B fname_encoding .br This ext4 feature provides file system level character encoding support -for files and directories name. This feature is name-preserving on the -disk, but it allows applications to lookup for a file in the file system -using any encoding equivalent version of the file name. - -This feature is required to perform in-kernel case-insensitive file -name lookups. +for directories with the casefold (+F) flag enabled. This feature is +name-preserving on the disk, but it allows applications to lookup for a +file in the file system using an encoding equivalent version of the file +name. .TP .B has_journal .br
Signature · 1 line
2.20.1
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 10/10] ext4.5.in: Document design changes on the casefold attribute
Quoted 4 lines · Gabriel Krisman Bertazi
Document the fact that the encoding support is only used by directories with the +F attribute.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted
A message that is not in the archive (sent off-list or not yet received): [email protected]
Theodore Ts'o
AppliedRe: [PATCH e2fsprogs 03/10] ext2fs: nls: Convert NFKD -> NFD
Quoted 4 lines · Gabriel Krisman Bertazi
Following the kernel transition, convert the normalization format from NFKD to NFD. This also regenerates the
utf8data.hdatabase.Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Thanks, applied.
- Ted