Unicode/UTF-8 support for XFS
Subsystems (2)
Hi,
I'm posting this RFC on Olaf's behalf, as he is busy with other projects.
First is a series of kernel patches, then a series of patches for xfsprogs, and then a test.
Note that I have removed the unicode database files prior to posting due to their large size. There are instructions on how to download them in the relevant commit headers.
Thanks,
Ben
Here are some notes of introduction from Olaf:
-----------------------------------------------------------------------------
Unicode/UTF-8 support for XFS
So we had a customer request proper unicode support...
Design notes.
XFS uses byte strings for filenames, so UTF-8 is the expected format for unicode filenames. This does raise the question what criteria a byte string must meet to be UTF-8. We settled on the following:
- Valid unicode code points are 0..0x10FFFF, except that
- The surrogates 0xD800..0xDFFF are not valid code points, and
- Valid UTF-8 must be a shortest encoding of a valid unicode code point.
In addition, U+0 (ASCII NUL, '\0') is used to terminate byte strings (and is itself not part of the string). Moreover strings may be length-limited in addition to being NUL-terminated (there is no such thing as an embedded NUL in a length-limited string).
Based on feedback on the earlier patches for unicode/UTF-8 support, we decided that a filename that does not match the above criteria should be treated as a binary blob, as opposed to being rejected. To stress: if any part of the string isn't valid UTF-8, then the entire string is treated as a binary blob. This matters once normalization is considered.
When comparing unicode strings for equality, normalization comes into play:
we must compare the normalized forms of strings, not just the raw sequences of bytes. There are a number of defined normalization forms for unicode. We decided on a variant of NFKD we call NFKDI. NFD was chosed over NFC, because calculating NFC requires calculating NFD first, followed by an additional step. NFKD was chosen over NFD because this makes filenames that ought to be equal compare as equal. My favorite example is the ways "office" can be spelled, when "fi" or "ffi" ligatures are used. NFKDI adds one more step of NFKD, in that it eliminates the code points that have the Default_Ignorable_Code_Point property from the comparison. These code points are as a rule invisible, but might (or might not) be pulled in when you copy/paste a string to be used as a filename. An example of these is U+00AD SOFT HYPHEN, a code point that only shows up if a word is split across lines.
If a filename is considered to be binary blob, comparison is based on a simple binary match. Normalization does not apply to any part of a blob.
The code uses ("leverages", in corp-speak) the existing infrastructure for case-insensitive filenames. Like the CI code, the name used to create a file is stored on disk, and returned in a lookup. When comparing filenames the normalized forms of the names being compared are generated on the fly from the non-normalized forms stored on disk.
If the borgbit (the bit enabling legacy ASCII-based CI) is set in the superblock, then case folding is added into the mix. This normalization form we call NFKDICF. It allows for the creation of case-insensitive filesystems with UTF-8 support.
-----------------------------------------------------------------------------
Implementation notes.
Strings are normalized using a trie that stores the relevant information. The trie itself is part of the XFS module, and about 250kB in size. The trie is not checked in: instead we add the source files from the Unicode Character Database and a program that creates the header containing the trie.
The key for a lookup in the trie is a UTF-8 sequence. Each valid UTF-8 sequence leads to a leaf. No invalid sequence does. This means that trie lookups can be used to validate UTF-8 sequences, which why there is no specialized code for the same purpose.
The trie contains information for the version of unicode in which each code point was defined. This matters because non-normalized strings are stored on disk, and newer versions of unicode may introduce new normalized forms. Ideally, the version of unicode used by the filesystem is stored in the filesystem.
The trie also accounts for corrections made in the past to normalizations. This has little value today, because any newly created filesystem would be using unicode version 7.0.0. It is included in order to show, not tell, that such corrections can be handled if they are added in future revisions.
The algorithm used to calculate the sequences of bytes for the normalized form of a UTF-8 string is tricky. The core is found in utf8byte(), with an explanation in the preceeding comment.
The non-XFS-specific supporting code is in separate source files, and be put in some other location in the Linux kernel source tree, if desired. These functions have the prefix 'utf8n' if they handle length-limited strings, and 'utf8' if they handle NUL-terminated strings.
-----------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/9 xfs: return the first match during case-insensitive lookup.
From: Olaf Weber <[email protected]>
Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.
--- fs/xfs/libxfs/xfs_dir2_block.c | 17 +++------ fs/xfs/libxfs/xfs_dir2_leaf.c | 37 ++++---------------- fs/xfs/libxfs/xfs_dir2_node.c | 79 ++++++++++++++++-------------------------- fs/xfs/libxfs/xfs_dir2_sf.c | 8 ++--- 4 files changed, 45 insertions(+), 96 deletions(-)
fs/xfs/libxfs/xfs_dir2_block.c +5 -12
@@ -725,28 +725,21 @@ xfs_dir2_block_lookup_int( dep = (xfs_dir2_data_entry_t *) ((char *)hdr +xfs_dir2_dataptr_to_off (args->geo, addr));/* -* Compare name and if it's an exact match, return the index -* and buffer. If it's the first case-insensitive match, store -* the index and buffer and continue looking for an exact match. +* Compare name and if it's a match, return the +* index and buffer. */ cmp = mp->m_dirnameops->compname (args, dep->name, dep->namelen); -if (cmp !=XFS_CMP_DIFFERENT && cmp != args->cmpresult) { +if (cmp !=XFS_CMP_DIFFERENT ) { args->cmpresult = cmp; *bpp = bp; *entno = mid; -if (cmp ==XFS_CMP_EXACT ) -return 0 ; +return 0 ; } }while (++mid <be32_to_cpu (btp->count) &&be32_to_cpu (blp[mid].hashval) == hash);ASSERT (args->op_flags &XFS_DA_OP_OKNOENT ); -/* -* Here, we can only be doing a lookup (not a rename or replace). -* If a case-insensitive match was found earlier, return success. -*/ -if (args->cmpresult ==XFS_CMP_CASE ) -return 0 ; +ASSERT (args->cmpresult ==XFS_CMP_DIFFERENT );/* * No match, release the buffer and return ENOENT. */
fs/xfs/libxfs/xfs_dir2_leaf.c +6 -31
@@ -1226,7 +1226,6 @@ xfs_dir2_leaf_lookup_int(xfs_mount_t *mp;/* filesystem mount point */ xfs_dir2_db_t newdb;/* new data block number */ xfs_trans_t *tp;/* transaction pointer */ -xfs_dir2_db_t cidb =-1 ;/* case match data block no. */ enum xfs_dacmp cmp;/* name compare result */ struct xfs_dir2_leaf_entry *ents;struct xfs_dir3_icleaf_hdr leafhdr;
@@ -1290,46 +1289,22 @@ xfs_dir2_leaf_lookup_int( be32_to_cpu(lep->address)));/* * Compare name and if it's an exact match, return the index -* and buffer. If it's the first case-insensitive match, store -* the index and buffer and continue looking for an exact match. +* and buffer */ cmp = mp->m_dirnameops->compname (args, dep->name, dep->namelen); -if (cmp !=XFS_CMP_DIFFERENT && cmp != args->cmpresult) { +if (cmp !=XFS_CMP_DIFFERENT ) { args->cmpresult = cmp; *indexp = index; -/* case exact match: return the current buffer. */ -if (cmp ==XFS_CMP_EXACT ) { - *dbpp = dbp; -return 0 ; - } - cidb = curdb; + *dbpp = dbp; +return 0 ; } }ASSERT (args->op_flags &XFS_DA_OP_OKNOENT ); -/* -* Here, we can only be doing a lookup (not a rename or remove). -* If a case-insensitive match was found earlier, re-read the -* appropriate data block if required and return it. -*/ -if (args->cmpresult ==XFS_CMP_CASE ) { -ASSERT (cidb !=-1 ); -if (cidb != curdb) { -xfs_trans_brelse (tp, dbp); - error =xfs_dir3_data_read (tp, dp, -xfs_dir2_db_to_da (args->geo, cidb), --1 , &dbp); -if (error) { -xfs_trans_brelse (tp, lbp); -return error; - } - } - *dbpp = dbp; -return 0 ; - } +ASSERT (args->cmpresult ==XFS_CMP_DIFFERENT ); +/* * No match found, return -ENOENT. */ -ASSERT (cidb ==-1 );if (dbp)xfs_trans_brelse (tp, dbp);xfs_trans_brelse (tp, lbp);
fs/xfs/libxfs/xfs_dir2_node.c +30 -49
123 lines not on this page · raw message
fs/xfs/libxfs/xfs_dir2_sf.c +4 -4
@@ -757,19 +757,19 @@ xfs_dir2_sf_lookup(for (i =0 , sfep =xfs_dir2_sf_firstentry (sfp); i < sfp->count; i++, sfep = dp->d_ops->sf_nextentry (sfp, sfep)) {/* -* Compare name and if it's an exact match, return the inode -* number. If it's the first case-insensitive match, store the -* inode number and continue looking for an exact match. +* Compare name and if it's a match, return the inode +* number. */ cmp = dp->i_mount->m_dirnameops->compname (args, sfep->name, sfep->namelen); -if (cmp !=XFS_CMP_DIFFERENT && cmp != args->cmpresult) { +if (cmp !=XFS_CMP_DIFFERENT ) { args->cmpresult = cmp; args->inumber = dp->d_ops->sf_get_ino (sfp, sfep); args->filetype = dp->d_ops->sf_get_ftype (sfep);if (cmp ==XFS_CMP_EXACT )return -EEXIST ; ci_sfep = sfep; +break ; } }ASSERT (args->op_flags &XFS_DA_OP_OKNOENT );
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/9 xfs: rename XFS_CMP_CASE to XFS_CMP_MATCH
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Rename XFS_CMP_CASE to XFS_CMP_MATCH. With unicode filenames and normalization, different strings will match on other criteria than case insensitivity.
--- fs/xfs/libxfs/xfs_da_btree.h | 2 +- fs/xfs/libxfs/xfs_dir2.c | 9 ++++++--- fs/xfs/libxfs/xfs_dir2_node.c | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_da_btree.h +1 -1
@@ -52,7 +52,7 @@ struct xfs_da_geometry {enum xfs_dacmp {XFS_CMP_DIFFERENT ,/* names are completely different */ XFS_CMP_EXACT ,/* names are exactly the same */ -XFS_CMP_CASE /* names are same but differ in case */ +XFS_CMP_MATCH /* names are same but differ in encoding */ };/*
fs/xfs/libxfs/xfs_dir2.c +6 -3
@@ -74,7 +74,7 @@ xfs_ascii_ci_compname(continue ;if (tolower (args->name[i]) !=tolower (name[i]))return XFS_CMP_DIFFERENT ; - result =XFS_CMP_CASE ; + result =XFS_CMP_MATCH ; }return result;
@@ -315,8 +315,11 @@ xfs_dir_cilookup_result( {if (args->cmpresult ==XFS_CMP_DIFFERENT )return -ENOENT ; -if (args->cmpresult !=XFS_CMP_CASE || - !(args->op_flags &XFS_DA_OP_CILOOKUP )) +if (args->cmpresult ==XFS_CMP_EXACT ) +return -EEXIST ; +ASSERT (args->cmpresult ==XFS_CMP_MATCH ); +/* Only dup the found name if XFS_DA_OP_CILOOKUP is set. */ +if (!(args->op_flags &XFS_DA_OP_CILOOKUP ))return -EEXIST ; args->value =kmem_alloc (len,KM_NOFS |KM_MAYFAIL );
fs/xfs/libxfs/xfs_dir2_node.c +1 -1
@@ -2023,7 +2023,7 @@ xfs_dir2_node_lookup( error =xfs_da3_node_lookup_int (state, &rval);if (error) rval = error; -else if (rval == -ENOENT && args->cmpresult ==XFS_CMP_CASE ) { +else if (rval == -ENOENT && args->cmpresult ==XFS_CMP_MATCH ) {/* If a CI match, dup the actual name and return -EEXIST */ xfs_dir2_data_entry_t *dep;
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/9 xfs: add xfs_nameops.normhash
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Add a normhash callout to the xfs_nameops. This callout takes an xfs_da_args structure as its argument, and calculates a hash value over the name. It may in the process create a normalized form of the name, and assign that to the norm/normlen fields in the xfs_da_args structure.
--- fs/xfs/libxfs/xfs_da_btree.c | 9 +++++++++ fs/xfs/libxfs/xfs_da_btree.h | 3 +++ fs/xfs/libxfs/xfs_dir2.c | 42 +++++++++++++++++++++++++++++++++++++----- 3 files changed, 49 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_da_btree.c +9 -0
@@ -1990,8 +1990,17 @@ xfs_default_hashname(return xfs_da_hashname (name->name, name->len); } +STATIC int +xfs_da_normhash ( +struct xfs_da_args *args) +{ + args->hashval =xfs_da_hashname (args->name, args->namelen); +return 0 ; +} +const struct xfs_nameops xfs_default_nameops = { .hashname = xfs_default_hashname, + .normhash = xfs_da_normhash, .compname = xfs_da_compname };
fs/xfs/libxfs/xfs_da_btree.h +3 -0
@@ -61,7 +61,9 @@ enum xfs_dacmp { typedef struct xfs_da_args { struct xfs_da_geometry *geo; /* da block geometry */ const __uint8_t *name; /* string (maybe not NULL terminated) */ + const __uint8_t *norm; /* normalized name (may be NULL) */ int namelen; /* length of string (maybe no NULL) */ + int normlen; /* length of normalized name */ __uint8_t filetype; /* filetype of inode for directories */ __uint8_t *value; /* set of bytes (maybe contain NULLs) */ int valuelen; /* length of value */
@@ -150,6 +152,7 @@ typedef struct xfs_da_state {*/ struct xfs_nameops {xfs_dahash_t (*hashname)(struct xfs_name *); +int (*normhash)(struct xfs_da_args *);enum xfs_dacmp (*compname)(struct xfs_da_args *,const unsigned char *,int ); };
fs/xfs/libxfs/xfs_dir2.c +37 -5
143 lines not on this page · raw message
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 4/9 xfs: change interface of xfs_nameops.normhash
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
With the introduction of the xfs_nameops.normhash callout, all uses of the hashname callout now occur in places where an xfs_name structure must be explicitly created just to match the parameter passing convention of this callout. Change the arguments to a const unsigned char * and int instead.
--- fs/xfs/libxfs/xfs_da_btree.c | 9 +-------- fs/xfs/libxfs/xfs_da_btree.h | 2 +- fs/xfs/libxfs/xfs_dir2.c | 7 ++++--- fs/xfs/libxfs/xfs_dir2_block.c | 2 +- fs/xfs/libxfs/xfs_dir2_data.c | 3 ++- 5 files changed, 9 insertions(+), 14 deletions(-)
fs/xfs/libxfs/xfs_da_btree.c +1 -8
@@ -1983,13 +1983,6 @@ xfs_da_compname( XFS_CMP_EXACT :XFS_CMP_DIFFERENT ; } -static xfs_dahash_t -xfs_default_hashname ( -struct xfs_name *name) -{ -return xfs_da_hashname (name->name, name->len); -} - STATIC intxfs_da_normhash (struct xfs_da_args *args)
@@ -1999,7 +1992,7 @@ xfs_da_normhash( }const struct xfs_nameops xfs_default_nameops = { - .hashname = xfs_default_hashname, + .hashname = xfs_da_hashname, .normhash = xfs_da_normhash, .compname = xfs_da_compname };
fs/xfs/libxfs/xfs_da_btree.h +1 -1
@@ -151,7 +151,7 @@ typedef struct xfs_da_state {* Name ops for directory and/or attr name operations */ struct xfs_nameops { -xfs_dahash_t (*hashname)(struct xfs_name *); +xfs_dahash_t (*hashname)(const unsigned char *,int );int (*normhash)(struct xfs_da_args *);enum xfs_dacmp (*compname)(struct xfs_da_args *,const unsigned char *,int );
fs/xfs/libxfs/xfs_dir2.c +4 -3
@@ -45,13 +45,14 @@ struct xfs_name xfs_name_dotdot = { (unsigned char *)"..", 2, XFS_DIR3_FT_DIR };*/ STATIC xfs_dahash_txfs_ascii_ci_hashname ( -struct xfs_name *name) +const unsigned char *name, +int len) {xfs_dahash_t hash;int i; -for (i =0 , hash =0 ; i < name->len; i++) - hash =tolower (name->name[i]) ^rol32 (hash,7 ); +for (i =0 , hash =0 ; i < len; i++) + hash =tolower (name[i]) ^rol32 (hash,7 );return hash; }
fs/xfs/libxfs/xfs_dir2_block.c +1 -1
@@ -1231,7 +1231,7 @@ xfs_dir2_sf_to_block( name.name = sfep->name; name.len = sfep->namelen; blp[2 + i].hashval =cpu_to_be32 (mp->m_dirnameops-> -hashname (&name)); +hashname (sfep->name, sfep->namelen)); blp[2 + i].address =cpu_to_be32 (xfs_dir2_byte_to_dataptr ( (char *)dep - (char *)hdr)); offset = (int )((char *)(tagp +1 ) - (char *)hdr);
fs/xfs/libxfs/xfs_dir2_data.c +2 -1
@@ -179,7 +179,8 @@ __xfs_dir3_data_check( ((char *)dep - (char *)hdr)); name.name = dep->name; name.len = dep->namelen; - hash = mp->m_dirnameops->hashname (&name); + hash = mp->m_dirnameops->hashname (dep->name, + dep->namelen);for (i =0 ; i <be32_to_cpu (btp->count); i++) {if (be32_to_cpu (lep[i].address) == addr &&be32_to_cpu (lep[i].hashval) == hash)
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 5/9 xfs: add a superblock feature bit to indicate UTF-8 support.
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
When UTF-8 support is enabled, the xfs_dir_ci_inode_operations must be installed. Add xfs_sb_version_hasci(), which tests both the borgbit and the utf8bit, and returns true if at least one of them is set. Replace calls to xfs_sb_version_hasasciici() as needed.
--- fs/xfs/libxfs/xfs_sb.h | 24 +++++++++++++++++++++++- fs/xfs/xfs_fs.h | 1 + fs/xfs/xfs_fsops.c | 4 +++- fs/xfs/xfs_iops.c | 4 ++-- 4 files changed, 29 insertions(+), 4 deletions(-)
fs/xfs/libxfs/xfs_sb.h +23 -1
@@ -70,6 +70,7 @@ struct xfs_trans;#define XFS_SB_VERSION2_RESERVED4BIT 0x00000004#define XFS_SB_VERSION2_ATTR2BIT 0x00000008/* Inline attr rework */ #define XFS_SB_VERSION2_PARENTBIT 0x00000010/* parent pointers */ +#define XFS_SB_VERSION2_UTF8BIT 0x00000020/* utf8 names */ #define XFS_SB_VERSION2_PROJID32BIT 0x00000080/* 32 bit project id */ #define XFS_SB_VERSION2_CRCBIT 0x00000100/* metadata CRCs */ #define XFS_SB_VERSION2_FTYPE 0x00000200/* inode type in dir */
@@ -77,6 +78,7 @@ struct xfs_trans;#define XFS_SB_VERSION2_OKBITS \ (XFS_SB_VERSION2_LAZYSBCOUNTBIT | \ XFS_SB_VERSION2_ATTR2BIT | \ + XFS_SB_VERSION2_UTF8BIT | \ XFS_SB_VERSION2_PROJID32BIT | \ XFS_SB_VERSION2_FTYPE)
@@ -509,8 +511,10 @@ xfs_sb_has_ro_compat_feature( }#define XFS_SB_FEAT_INCOMPAT_FTYPE (1 << 0)/* filetype in dirent */ +#define XFS_SB_FEAT_INCOMPAT_UTF8 (1 << 1)/* utf-8 name support */ #define XFS_SB_FEAT_INCOMPAT_ALL \ - (XFS_SB_FEAT_INCOMPAT_FTYPE) + (XFS_SB_FEAT_INCOMPAT_FTYPE | \ + XFS_SB_FEAT_INCOMPAT_UTF8)#define XFS_SB_FEAT_INCOMPAT_UNKNOWN ~XFS_SB_FEAT_INCOMPAT_ALL static inline bool
@@ -558,6 +562,24 @@ static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp) (sbp->sb_features_ro_compat &XFS_SB_FEAT_RO_COMPAT_FINOBT ); } +static inline int xfs_sb_version_hasutf8 (xfs_sb_t *sbp) +{ +return (XFS_SB_VERSION_NUM (sbp) ==XFS_SB_VERSION_5 && +xfs_sb_has_incompat_feature (sbp,XFS_SB_FEAT_INCOMPAT_UTF8 )) || + (xfs_sb_version_hasmorebits (sbp) && + (sbp->sb_features2 &XFS_SB_VERSION2_UTF8BIT )); +} + +/* +* Special case: there are a number of places where we need to test +* both the borgbit and the utf8bit, and take the same action if +* either of those is set. +*/ +static inline int xfs_sb_version_hasci (xfs_sb_t *sbp) +{ +return xfs_sb_version_hasasciici (sbp) ||xfs_sb_version_hasutf8 (sbp); +} +/* * end of superblock version macros */
fs/xfs/xfs_fs.h +1 -0
@@ -239,6 +239,7 @@ typedef struct xfs_fsop_resblks {#define XFS_FSOP_GEOM_FLAGS_V5SB 0x8000/* version 5 superblock */ #define XFS_FSOP_GEOM_FLAGS_FTYPE 0x10000/* inode directory types */ #define XFS_FSOP_GEOM_FLAGS_FINOBT 0x20000/* free inode btree */ +#define XFS_FSOP_GEOM_FLAGS_UTF8 0x40000/* utf8 filenames */ /* * Minimum and maximum sizes need for growth checks.
fs/xfs/xfs_fsops.c +3 -1
@@ -103,7 +103,9 @@ xfs_fs_geometry( (xfs_sb_version_hasftype (&mp->m_sb) ?XFS_FSOP_GEOM_FLAGS_FTYPE :0 ) | (xfs_sb_version_hasfinobt (&mp->m_sb) ? -XFS_FSOP_GEOM_FLAGS_FINOBT :0 ); +XFS_FSOP_GEOM_FLAGS_FINOBT :0 ) | + (xfs_sb_version_hasutf8 (&mp->m_sb) ? +XFS_FSOP_GEOM_FLAGS_UTF8 :0 ); geo->logsectsize =xfs_sb_version_hassector (&mp->m_sb) ? mp->m_sb.sb_logsectsize :BBSIZE ; geo->rtsectsize = mp->m_sb.sb_blocksize;
fs/xfs/xfs_iops.c +2 -2
@@ -335,9 +335,9 @@ xfs_vn_unlink(/* * With unlink, the VFS makes the dentry "negative": no inode, * but still hashed. This is incompatible with case-insensitive -* mode, so invalidate (unhash) the dentry in CI-mode. +* or utf8 mode, so invalidate (unhash) the dentry in CI-mode. */ -if (xfs_sb_version_hasasciici (&XFS_M (dir->i_sb)->m_sb)) +if (xfs_sb_version_hasci (&XFS_M (dir->i_sb)->m_sb))d_invalidate (dentry);return 0 ; }
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 6/9 xfs: add unicode character database files
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Add files from the Unicode Character Database, version 7.0.0, to the source. A helper program that generates a trie used for normalization from these files is part of a separate commit.
---
[v2: Removed large unicode files prior to posting. Get them as below. -bpm]
cd fs/xfs/support/ucd
wget http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt
wget http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt
wget http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt
wget http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt
wget http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt
wget http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt
wget http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt
for e in *.txt
do
base=`basename $e .txt` mv $e $base-7.0.0.txt done --- fs/xfs/support/ucd/README | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 fs/xfs/support/ucd/README
fs/xfs/support/ucd/README new file +33 -0
@@ -0,0 +1,33 @@ +The files in this directory are part of the Unicode Character Database +for version 7.0.0 of the Unicode standard. + +The full set of files can be found here: + + http://www.unicode.org/Public/7.0.0/ucd/ + +The latest released version of the UCD can be found here: + + http://www.unicode.org/Public/UCD/latest/ + +The files in this directory are identical, except that they have been +renamed with a suffix indicating the unicode version. + +Individual source links: + + http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt + http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt + http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt + http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt + http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt + http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt + http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt + +md5sums + + 9a92b2bfe56c6719def926bab524fefd CaseFolding-7.0.0.txt + 07b8b1027eb824cf0835314e94f23d2e DerivedAge-7.0.0.txt + 90c3340b16821e2f2153acdbe6fc6180 DerivedCombiningClass-7.0.0.txt + c41c0601f808116f623de47110ed4f93 DerivedCoreProperties-7.0.0.txt + 522720ddfc150d8e63a2518634829bce NormalizationCorrections-7.0.0.txt + 1f35175eba4a2ad795db489f789ae352 NormalizationTest-7.0.0.txt + c8355655731d75e6a3de8c20d7e601ba UnicodeData-7.0.0.txt
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 7/9 xfs: add trie generator and supporting code for UTF-8.
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
mkutf8data.c is the source for a program that generates utf8data.h, which contains the trie that utf8norm.c uses. The trie is generated from the Unicode 7.0.0 data files. The format of the utf8data[] table is described in utf8norm.c.
Supporting functions for UTF-8 normalization are in utf8norm.c with the header utf8norm.h. Two normalization forms are supported: nfkdi and nfkdicf.
nfkdi:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
nfkdicf:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
- Apply a full casefold (C + F).
For the purposes of the code, a string is valid UTF-8 if:
- The values encoded are 0x1..0x10FFFF.
- The surrogate codepoints 0xD800..0xDFFFF are not encoded.
- The shortest possible encoding is used for all values.
The supporting functions work on null-terminated strings (utf8 prefix) and on length-limited strings (utf8n prefix).
--- fs/xfs/Makefile | 19 + fs/xfs/support/mkutf8data.c | 3239 +++++++++++++++++++++++++++++++++++++++++++ fs/xfs/support/utf8norm.c | 641 +++++++++ fs/xfs/support/utf8norm.h | 111 ++ 4 files changed, 4010 insertions(+) create mode 100644 fs/xfs/support/mkutf8data.c create mode 100644 fs/xfs/support/utf8norm.c create mode 100644 fs/xfs/support/utf8norm.h
fs/xfs/Makefile +19 -0
@@ -92,6 +92,25 @@ xfs-y += xfs_aops.o \ kmem.o \ uuid.o +# Objects in support/ +xfs-y += support/utf8norm.o + +hostprogs-y := support/mkutf8data +$(obj)/support/utf8norm.o: $(obj)/support/utf8data.h +$(obj)/support/utf8data.h: $(src)/support/ucd/*.txt +$(obj)/support/utf8data.h: $(obj)/support/mkutf8data FORCE + $(call if_changed,mkutf8data) +quiet_cmd_mkutf8data = MKUTF8DATA $@ + cmd_mkutf8data = $(obj)/support/mkutf8data \ + -a $(src)/support/ucd/DerivedAge-7.0.0.txt \ + -c $(src)/support/ucd/DerivedCombiningClass-7.0.0.txt \ + -p $(src)/support/ucd/DerivedCoreProperties-7.0.0.txt \ + -d $(src)/support/ucd/UnicodeData-7.0.0.txt \ + -f $(src)/support/ucd/CaseFolding-7.0.0.txt \ + -n $(src)/support/ucd/NormalizationCorrections-7.0.0.txt \ + -t $(src)/support/ucd/NormalizationTest-7.0.0.txt \ + -o $@ + # low-level transaction/log code xfs-y += xfs_log.o \ xfs_log_cil.o \
fs/xfs/support/mkutf8data.c new file +3239 -0
2,975 lines not on this page · raw message
fs/xfs/support/utf8norm.c new file +641 -0
641 lines not on this page · raw message
fs/xfs/support/utf8norm.h new file +111 -0
111 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Part of the diff is not shown here. View the raw message.
PATCH 8/9 xfs: add xfs_nameops for utf8 and utf8+casefold.
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
The xfs_utf8_nameops use the nfkdi normalization when comparing filenames, and are installed if the utf8bit is set in the super block.
The xfs_utf8_ci_nameops use the nfkdicf normalization when comparing filenames, and are installed if both the utf8bit and the borgbit are set in the superblock.
Normalized filenames are not stored on disk. Normalization will fail if a filename is not valid UTF-8, in which case the filename is treated as an opaque blob.
--- fs/xfs/Makefile | 1 + fs/xfs/libxfs/xfs_dir2.c | 16 +++- fs/xfs/xfs_iops.c | 2 +- fs/xfs/xfs_utf8.c | 242 +++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_utf8.h | 25 +++++ 5 files changed, 281 insertions(+), 5 deletions(-) create mode 100644 fs/xfs/xfs_utf8.c create mode 100644 fs/xfs/xfs_utf8.h
fs/xfs/Makefile +1 -0
@@ -88,6 +88,7 @@ xfs-y += xfs_aops.o \ xfs_symlink.o \ xfs_sysfs.o \ xfs_trans.o \ + xfs_utf8.o \ xfs_xattr.o \ kmem.o \ uuid.o
fs/xfs/libxfs/xfs_dir2.c +12 -4
@@ -35,6 +35,7 @@#include "xfs_error.h" #include "xfs_trace.h" #include "xfs_dinode.h" +#include "xfs_utf8.h" struct xfs_name xfs_name_dotdot = { (unsigned char *)".." ,2 ,XFS_DIR3_FT_DIR };
@@ -156,10 +157,17 @@ xfs_da_mount( (uint )sizeof (xfs_da_node_entry_t); dageo->magicpct = (dageo->blksize *37 ) /100 ; -if (xfs_sb_version_hasasciici (&mp->m_sb)) - mp->m_dirnameops = &xfs_ascii_ci_nameops; -else - mp->m_dirnameops = &xfs_default_nameops; +if (xfs_sb_version_hasutf8 (&mp->m_sb)) { +if (xfs_sb_version_hasasciici (&mp->m_sb)) + mp->m_dirnameops = &xfs_utf8_ci_nameops; +else + mp->m_dirnameops = &xfs_utf8_nameops; + }else { +if (xfs_sb_version_hasasciici (&mp->m_sb)) + mp->m_dirnameops = &xfs_ascii_ci_nameops; +else + mp->m_dirnameops = &xfs_default_nameops; + }return 0 ; }
fs/xfs/xfs_iops.c +1 -1
@@ -1257,7 +1257,7 @@ xfs_setup_inode(break ;case S_IFDIR :lockdep_set_class (&ip->i_lock.mr_lock, &xfs_dir_ilock_class); -if (xfs_sb_version_hasasciici (&XFS_M (inode->i_sb)->m_sb)) +if (xfs_sb_version_hasci (&XFS_M (inode->i_sb)->m_sb)) inode->i_op = &xfs_dir_ci_inode_operations;else inode->i_op = &xfs_dir_inode_operations;
fs/xfs/xfs_utf8.c new file +242 -0
242 lines not on this page · raw message
fs/xfs/xfs_utf8.h new file +25 -0
@@ -0,0 +1,25 @@ +/* +* Copyright (c) 2014 SGI. +* All rights reserved. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public License as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it would be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write the Free Software Foundation, +* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +*/ + +#ifndef XFS_UTF8_H +#define XFS_UTF8_H + +extern struct xfs_nameops xfs_utf8_nameops; +extern struct xfs_nameops xfs_utf8_ci_nameops; + +#endif /* XFS_UTF8_H */
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 9/9 xfs: apply utf-8 normalization rules to user extended attribute names
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Apply the same rules for UTF-8 normalization to the names of user-defined extended attributes. System attributes are excluded because they are not user-visible in the first place, and the kernel is expected to know what it is doing when naming them.
--- fs/xfs/libxfs/xfs_attr.c | 56 ++++++++++++++++++++++++++++++++++++------- fs/xfs/libxfs/xfs_attr_leaf.c | 11 +++++++-- fs/xfs/xfs_attr_list.c | 11 ++++++++- fs/xfs/xfs_utf8.c | 7 ++++++ 4 files changed, 74 insertions(+), 11 deletions(-)
fs/xfs/libxfs/xfs_attr.c +48 -8
@@ -83,12 +83,14 @@ xfs_attr_args_init( const unsigned char *name, int flags) { +struct xfs_mount *mp = dp->i_mount; +int error;if (!name)return -EINVAL ;memset (args,0 ,sizeof (*args)); - args->geo = dp->i_mount->m_attr_geo; + args->geo = mp->m_attr_geo; args->whichfork =XFS_ATTR_FORK ; args->dp = dp; args->flags = flags;
@@ -97,7 +99,11 @@ xfs_attr_args_init(if (args->namelen >=MAXNAMELEN )return -EFAULT ;/* match IRIX behaviour */ - args->hashval =xfs_da_hashname (args->name, args->namelen); +if (!xfs_sb_version_hasutf8 (&mp->m_sb)) + args->hashval =xfs_da_hashname (args->name, args->namelen); +else if ((error = mp->m_dirnameops->normhash (args)) !=0 ) +return error; +return 0 ; }
@@ -154,6 +160,9 @@ xfs_attr_get( error =xfs_attr_node_get (&args);xfs_iunlock (ip, lock_mode); +if (args.norm) +kmem_free (args.norm); + *valuelenp = args.valuelen;return error == -EEXIST ?0 : error; }
@@ -216,8 +225,11 @@ xfs_attr_set(return -EIO ; error =xfs_attr_args_init (&args, dp, name, flags); -if (error) +if (error) { +if (args.norm) +kmem_free (args.norm);return error; + } args.value = value; args.valuelen = valuelen;
@@ -227,8 +239,11 @@ xfs_attr_set( args.total =xfs_attr_calc_size (&args, &local); error =xfs_qm_dqattach (dp,0 ); -if (error) +if (error) { +if (args.norm) +kmem_free (args.norm);return error; + }/* * If the inode doesn't have an attribute fork, add one.
@@ -239,8 +254,11 @@ xfs_attr_set(XFS_ATTR_SF_ENTSIZE_BYNAME (args.namelen, valuelen); error =xfs_bmap_add_attrfork (dp, sf_size, rsvd); -if (error) +if (error) { +if (args.norm) +kmem_free (args.norm);return error; + } }/*
@@ -270,6 +288,8 @@ xfs_attr_set( error =xfs_trans_reserve (args.trans, &tres, args.total,0 );if (error) {xfs_trans_cancel (args.trans,0 ); +if (args.norm) +kmem_free (args.norm);return error; }xfs_ilock (dp,XFS_ILOCK_EXCL );
@@ -280,6 +300,8 @@ xfs_attr_set(if (error) {xfs_iunlock (dp,XFS_ILOCK_EXCL );xfs_trans_cancel (args.trans,XFS_TRANS_RELEASE_LOG_RES ); +if (args.norm) +kmem_free (args.norm);return error; }
@@ -327,6 +349,8 @@ xfs_attr_set( XFS_TRANS_RELEASE_LOG_RES);xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free (args.norm);return error ? error : err2; }
@@ -388,7 +412,8 @@ xfs_attr_set(xfs_trans_log_inode (args.trans, dp,XFS_ILOG_CORE ); error =xfs_trans_commit (args.trans,XFS_TRANS_RELEASE_LOG_RES );xfs_iunlock (dp,XFS_ILOCK_EXCL ); - +if (args.norm) +kmem_free (args.norm);return error; out:
@@ -397,6 +422,8 @@ out: XFS_TRANS_RELEASE_LOG_RES|XFS_TRANS_ABORT); }xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free (args.norm);return error; }
@@ -425,8 +452,11 @@ xfs_attr_remove(return -ENOATTR ; error =xfs_attr_args_init (&args, dp, name, flags); -if (error) +if (error) { +if (args.norm) +kmem_free (args.norm);return error; + } args.firstblock = &firstblock; args.flist = &flist;
@@ -439,8 +469,11 @@ xfs_attr_remove( args.op_flags =XFS_DA_OP_OKNOENT ; error =xfs_qm_dqattach (dp,0 ); -if (error) +if (error) { +if (args.norm) +kmem_free (args.norm);return error; + }/* * Start our first transaction of the day.
@@ -466,6 +499,8 @@ xfs_attr_remove( XFS_ATTRRM_SPACE_RES(mp), 0);if (error) {xfs_trans_cancel (args.trans,0 ); +if (args.norm) +kmem_free (args.norm);return error; }
@@ -506,6 +541,8 @@ xfs_attr_remove(xfs_trans_log_inode (args.trans, dp,XFS_ILOG_CORE ); error =xfs_trans_commit (args.trans,XFS_TRANS_RELEASE_LOG_RES );xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free (args.norm);return error;
@@ -515,6 +552,9 @@ out: XFS_TRANS_RELEASE_LOG_RES|XFS_TRANS_ABORT); }xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free (args.norm); +return error; }
fs/xfs/libxfs/xfs_attr_leaf.c +9 -2
@@ -661,6 +661,7 @@ int xfs_attr_shortform_to_leaf(xfs_da_args_t *args) {xfs_inode_t *dp; +struct xfs_mount *mp;xfs_attr_shortform_t *sf;xfs_attr_sf_entry_t *sfe;xfs_da_args_t nargs;
@@ -673,6 +674,7 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args)trace_xfs_attr_sf_to_leaf (args); dp = args->dp; + mp = dp->i_mount; ifp = dp->i_afp; sf = (xfs_attr_shortform_t *)ifp->if_u1.if_data; size =be16_to_cpu (sf->hdr.totsize);
@@ -726,13 +728,18 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args) nargs.namelen = sfe->namelen; nargs.value = &sfe->nameval[nargs.namelen]; nargs.valuelen = sfe->valuelen; - nargs.hashval =xfs_da_hashname (sfe->nameval, - sfe->namelen); nargs.flags =XFS_ATTR_NSP_ONDISK_TO_ARGS (sfe->flags); +if (!xfs_sb_version_hasutf8 (&mp->m_sb)) + nargs.hashval =xfs_da_hashname (sfe->nameval, + sfe->namelen); +else if ((error = mp->m_dirnameops->normhash (&nargs)) !=0 ) + goto out; error =xfs_attr3_leaf_lookup_int (bp, &nargs);/* set a->index */ ASSERT (error == -ENOATTR ); error =xfs_attr3_leaf_add (bp, &nargs);ASSERT (error != -ENOSPC ); +if (nargs.norm) +kmem_free (nargs.norm);if (error) goto out; sfe =XFS_ATTR_SF_NEXTENTRY (sfe);
fs/xfs/xfs_attr_list.c +10 -1
@@ -76,12 +76,14 @@ xfs_attr_shortform_list(xfs_attr_list_context_t *context)xfs_attr_shortform_t *sf;xfs_attr_sf_entry_t *sfe;xfs_inode_t *dp; +struct xfs_mount *mp;int sbsize, nsbuf, count, i;int error;ASSERT (context !=NULL ); dp = context->dp;ASSERT (dp !=NULL ); + mp = dp->i_mount;ASSERT (dp->i_afp !=NULL ); sf = (xfs_attr_shortform_t *)dp->i_afp->if_u1.if_data;ASSERT (sf !=NULL );
@@ -154,7 +156,14 @@ xfs_attr_shortform_list(xfs_attr_list_context_t *context) } sbp->entno = i; - sbp->hash =xfs_da_hashname (sfe->nameval, sfe->namelen); +/* ATTR_ROOT and ATTR_SECURE are never normalized. */ +if (!xfs_sb_version_hasutf8 (&mp->m_sb) || + (sfe->flags & (ATTR_ROOT |ATTR_SECURE ))) { + sbp->hash =xfs_da_hashname (sfe->nameval, sfe->namelen); + }else { + sbp->hash = mp->m_dirnameops->hashname (sfe->nameval, + sfe->namelen); + } sbp->name = sfe->nameval; sbp->namelen = sfe->namelen;/* These are bytes, and both on-disk, don't endian-flip */
fs/xfs/xfs_utf8.c +7 -0
@@ -38,6 +38,7 @@#include "xfs_inode.h" #include "xfs_inode_item.h" #include "xfs_bmap.h" +#include "xfs_attr.h" #include "xfs_error.h" #include "xfs_trace.h" #include "xfs_utf8.h"
@@ -80,6 +81,9 @@ xfs_utf8_normhash(ssize_t normlen;int c; +/* Don't normalize system attribute names. */ +if (args->flags & (ATTR_ROOT |ATTR_SECURE )) + goto blob; nfkdi =utf8nfkdi (utf8version);/* Failure to normalize is treated as a blob. */ if ((normlen =utf8nlen (nfkdi, args->name, args->namelen)) <0 )
@@ -179,6 +183,9 @@ xfs_utf8_ci_normhash(ssize_t normlen;int c; +/* Don't normalize system attribute names. */ +if (args->flags & (ATTR_ROOT |ATTR_SECURE )) + goto blob; nfkdicf =utf8nfkdicf (utf8version);/* Failure to normalize is treated as a blob. */ if ((normlen =utf8nlen (nfkdicf, args->name, args->namelen)) <0 )
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/13 libxfs: return the first match during case-insensitive lookup
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.
--- libxfs/xfs_dir2_block.c | 17 ++++------- libxfs/xfs_dir2_leaf.c | 38 ++++------------------- libxfs/xfs_dir2_node.c | 80 ++++++++++++++++++------------------------------- libxfs/xfs_dir2_sf.c | 8 ++--- 4 files changed, 44 insertions(+), 99 deletions(-)
libxfs/xfs_dir2_block.c +5 -12
@@ -705,28 +705,21 @@ xfs_dir2_block_lookup_int( dep = (xfs_dir2_data_entry_t *) ((char *)hdr +xfs_dir2_dataptr_to_off (mp, addr));/* -* Compare name and if it's an exact match, return the index -* and buffer. If it's the first case-insensitive match, store -* the index and buffer and continue looking for an exact match. +* Compare name and if it's a match, return the +* index and buffer. */ cmp = mp->m_dirnameops->compname (args, dep->name, dep->namelen); -if (cmp !=XFS_CMP_DIFFERENT && cmp != args->cmpresult) { +if (cmp !=XFS_CMP_DIFFERENT ) { args->cmpresult = cmp; *bpp = bp; *entno = mid; -if (cmp ==XFS_CMP_EXACT ) -return 0 ; +return 0 ; } }while (++mid <be32_to_cpu (btp->count) &&be32_to_cpu (blp[mid].hashval) == hash);ASSERT (args->op_flags &XFS_DA_OP_OKNOENT ); -/* -* Here, we can only be doing a lookup (not a rename or replace). -* If a case-insensitive match was found earlier, return success. -*/ -if (args->cmpresult ==XFS_CMP_CASE ) -return 0 ; +ASSERT (args->cmpresult ==XFS_CMP_DIFFERENT );/* * No match, release the buffer and return ENOENT. */
libxfs/xfs_dir2_leaf.c +6 -32
@@ -1246,7 +1246,6 @@ xfs_dir2_leaf_lookup_int(xfs_mount_t *mp;/* filesystem mount point */ xfs_dir2_db_t newdb;/* new data block number */ xfs_trans_t *tp;/* transaction pointer */ -xfs_dir2_db_t cidb =-1 ;/* case match data block no. */ enum xfs_dacmp cmp;/* name compare result */ struct xfs_dir2_leaf_entry *ents;struct xfs_dir3_icleaf_hdr leafhdr;
@@ -1307,47 +1306,22 @@ xfs_dir2_leaf_lookup_int( dep = (xfs_dir2_data_entry_t *)((char *)dbp->b_addr +xfs_dir2_dataptr_to_off (mp,be32_to_cpu (lep->address)));/* -* Compare name and if it's an exact match, return the index -* and buffer. If it's the first case-insensitive match, store -* the index and buffer and continue looking for an exact match. +* Compare name and if it's a match, return the index +* and buffer. */ cmp = mp->m_dirnameops->compname (args, dep->name, dep->namelen); -if (cmp !=XFS_CMP_DIFFERENT && cmp != args->cmpresult) { +if (cmp !=XFS_CMP_DIFFERENT ) { args->cmpresult = cmp; *indexp = index; -/* case exact match: return the current buffer. */ -if (cmp ==XFS_CMP_EXACT ) { - *dbpp = dbp; -return 0 ; - } - cidb = curdb; + *dbpp = dbp; +return 0 ; } }ASSERT (args->op_flags &XFS_DA_OP_OKNOENT ); -/* -* Here, we can only be doing a lookup (not a rename or remove). -* If a case-insensitive match was found earlier, re-read the -* appropriate data block if required and return it. -*/ -if (args->cmpresult ==XFS_CMP_CASE ) { -ASSERT (cidb !=-1 ); -if (cidb != curdb) { -xfs_trans_brelse (tp, dbp); - error =xfs_dir3_data_read (tp, dp, -xfs_dir2_db_to_da (mp, cidb), --1 , &dbp); -if (error) { -xfs_trans_brelse (tp, lbp); -return error; - } - } - *dbpp = dbp; -return 0 ; - } +ASSERT (args->cmpresult ==XFS_CMP_DIFFERENT );/* * No match found, return ENOENT. */ -ASSERT (cidb ==-1 );if (dbp)xfs_trans_brelse (tp, dbp);xfs_trans_brelse (tp, lbp);
libxfs/xfs_dir2_node.c +29 -51
121 lines not on this page · raw message
libxfs/xfs_dir2_sf.c +4 -4
@@ -833,13 +833,12 @@ xfs_dir2_sf_lookup(for (i =0 , sfep =xfs_dir2_sf_firstentry (sfp); i < sfp->count; i++, sfep =xfs_dir3_sf_nextentry (dp->i_mount, sfp, sfep)) {/* -* Compare name and if it's an exact match, return the inode -* number. If it's the first case-insensitive match, store the -* inode number and continue looking for an exact match. +* Compare name and if it's a match, return the inode +* number. */ cmp = dp->i_mount->m_dirnameops->compname (args, sfep->name, sfep->namelen); -if (cmp !=XFS_CMP_DIFFERENT && cmp != args->cmpresult) { +if (cmp !=XFS_CMP_DIFFERENT ) { args->cmpresult = cmp; args->inumber =xfs_dir3_sfe_get_ino (dp->i_mount, sfp, sfep);
@@ -848,6 +847,7 @@ xfs_dir2_sf_lookup(if (cmp ==XFS_CMP_EXACT )return XFS_ERROR (EEXIST ); ci_sfep = sfep; +break ; } }ASSERT (args->op_flags &XFS_DA_OP_OKNOENT );
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/13 libxfs: rename XFS_CMP_CASE to XFS_CMP_MATCH
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Rename XFS_CMP_CASE to XFS_CMP_MATCH. With unicode filenames and normalization, different strings will match on other criteria than case insensitivity.
--- include/xfs_da_btree.h | 2 +- libxfs/xfs_dir2.c | 9 ++++++--- libxfs/xfs_dir2_node.c | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-)
include/xfs_da_btree.h +1 -1
@@ -34,7 +34,7 @@ struct zone;enum xfs_dacmp {XFS_CMP_DIFFERENT ,/* names are completely different */ XFS_CMP_EXACT ,/* names are exactly the same */ -XFS_CMP_CASE /* names are same but differ in case */ +XFS_CMP_MATCH /* names are same but differ in encoding */ };/*
libxfs/xfs_dir2.c +6 -3
@@ -72,7 +72,7 @@ xfs_ascii_ci_compname(continue ;if (tolower (args->name[i]) !=tolower (name[i]))return XFS_CMP_DIFFERENT ; - result =XFS_CMP_CASE ; + result =XFS_CMP_MATCH ; }return result;
@@ -248,8 +248,11 @@ xfs_dir_cilookup_result( {if (args->cmpresult ==XFS_CMP_DIFFERENT )return ENOENT ; -if (args->cmpresult !=XFS_CMP_CASE || - !(args->op_flags &XFS_DA_OP_CILOOKUP )) +if (args->cmpresult ==XFS_CMP_EXACT ) +return EEXIST ; +ASSERT (args->cmpresult ==XFS_CMP_MATCH ); +/* Only dup the found name if XFS_DA_OP_CILOOKUP is set. */ +if (!(args->op_flags &XFS_DA_OP_CILOOKUP ))return EEXIST ; args->value =kmem_alloc (len,KM_NOFS |KM_MAYFAIL );
libxfs/xfs_dir2_node.c +1 -1
@@ -2034,7 +2034,7 @@ xfs_dir2_node_lookup( error =xfs_da3_node_lookup_int (state, &rval);if (error) rval = error; -else if (rval ==ENOENT && args->cmpresult ==XFS_CMP_CASE ) { +else if (rval ==ENOENT && args->cmpresult ==XFS_CMP_MATCH ) {/* If a CI match, dup the actual name and return EEXIST */ xfs_dir2_data_entry_t *dep;
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/13 libxfs: add xfs_nameops.normhash
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Add a normhash callout to the xfs_nameops. This callout takes an xfs_da_args structure as its argument, and calculates a hash value over the name. It may in the process create a normalized form of the name, and assign that to the norm/normlen fields in the xfs_da_args structure.
Changes:
The pointer in kmem_free() was type converted to suppress compiler warnings.
--- include/xfs_da_btree.h | 5 ++++- libxfs/xfs_da_btree.c | 9 ++++++++ libxfs/xfs_dir2.c | 56 +++++++++++++++++++++++++++++++++++++++----------- 3 files changed, 57 insertions(+), 13 deletions(-)
include/xfs_da_btree.h +4 -1
@@ -42,7 +42,9 @@ enum xfs_dacmp {*/ typedef struct xfs_da_args { const __uint8_t *name; /* string (maybe not NULL terminated) */ - int namelen; /* length of string (maybe no NULL) */ + const __uint8_t *norm; /* normalized name (may be NULL) */ + int namelen; /* length of string (maybe no NULL) */ + int normlen; /* length of normalized name */ __uint8_t filetype; /* filetype of inode for directories */ __uint8_t *value; /* set of bytes (maybe contain NULLs) */ int valuelen; /* length of value */
@@ -131,6 +133,7 @@ typedef struct xfs_da_state {*/ struct xfs_nameops {xfs_dahash_t (*hashname)(struct xfs_name *); +int (*normhash)(struct xfs_da_args *);enum xfs_dacmp (*compname)(struct xfs_da_args *,const unsigned char *,int ); };
libxfs/xfs_da_btree.c +9 -0
@@ -2000,8 +2000,17 @@ xfs_default_hashname(return xfs_da_hashname (name->name, name->len); } +STATIC int +xfs_da_normhash ( +struct xfs_da_args *args) +{ + args->hashval =xfs_da_hashname (args->name, args->namelen); +return 0 ; +} +const struct xfs_nameops xfs_default_nameops = { .hashname = xfs_default_hashname, + .normhash = xfs_da_normhash, .compname = xfs_da_compname };
libxfs/xfs_dir2.c +44 -12
162 lines not on this page · raw message
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 4/13 libxfs: change interface of xfs_nameops.normhash
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
With the introduction of the xfs_nameops.normhash callout, all uses of the hashname callout now occur in places where an xfs_name structure must be explicitly created just to match the parameter passing convention of this callout. Change the arguments to a const unsigned char * and int instead.
--- db/check.c | 6 ++---- include/xfs_da_btree.h | 2 +- libxfs/xfs_da_btree.c | 9 +-------- libxfs/xfs_dir2.c | 10 ++++++---- libxfs/xfs_dir2_block.c | 5 +---- libxfs/xfs_dir2_data.c | 6 ++---- repair/phase6.c | 2 +- 7 files changed, 14 insertions(+), 26 deletions(-)
db/check.c +2 -4
@@ -2212,7 +2212,6 @@ process_data_dir_v2(int stale =0 ;int tag_err;__be16 *tagp; -struct xfs_name xname; data = iocur_top->data; block = iocur_top->data;
@@ -2323,9 +2322,8 @@ process_data_dir_v2( tag_err +=be16_to_cpu (*tagp) != (char *)dep - (char *)data; addr =xfs_dir2_db_off_to_dataptr (mp, db, (char *)dep - (char *)data); - xname.name = dep->name; - xname.len = dep->namelen; -dir_hash_add (mp->m_dirnameops->hashname (&xname), addr); +dir_hash_add (mp->m_dirnameops->hashname (dep->name, + dep->namelen), addr); ptr +=xfs_dir3_data_entsize (mp, dep->namelen); count++; lastfree =0 ;
include/xfs_da_btree.h +1 -1
@@ -132,7 +132,7 @@ typedef struct xfs_da_state {* Name ops for directory and/or attr name operations */ struct xfs_nameops { -xfs_dahash_t (*hashname)(struct xfs_name *); +xfs_dahash_t (*hashname)(const unsigned char *,int );int (*normhash)(struct xfs_da_args *);enum xfs_dacmp (*compname)(struct xfs_da_args *,const unsigned char *,int );
libxfs/xfs_da_btree.c +1 -8
@@ -1993,13 +1993,6 @@ xfs_da_compname( XFS_CMP_EXACT :XFS_CMP_DIFFERENT ; } -static xfs_dahash_t -xfs_default_hashname ( -struct xfs_name *name) -{ -return xfs_da_hashname (name->name, name->len); -} - STATIC intxfs_da_normhash (struct xfs_da_args *args)
@@ -2009,7 +2002,7 @@ xfs_da_normhash( }const struct xfs_nameops xfs_default_nameops = { - .hashname = xfs_default_hashname, + .hashname = xfs_da_hashname, .normhash = xfs_da_normhash, .compname = xfs_da_compname };
libxfs/xfs_dir2.c +6 -4
@@ -43,13 +43,14 @@ const unsigned char xfs_mode_to_ftype[S_IFMT >> S_SHIFT] = {*/ STATIC xfs_dahash_txfs_ascii_ci_hashname ( -struct xfs_name *name) +const unsigned char *name, +int len) {xfs_dahash_t hash;int i; -for (i =0 , hash =0 ; i < name->len; i++) - hash =tolower (name->name[i]) ^rol32 (hash,7 ); +for (i =0 , hash =0 ; i < len; i++) + hash =tolower (name[i]) ^rol32 (hash,7 );return hash; }
@@ -475,7 +476,8 @@ xfs_dir_canenter( args.name = name->name; args.namelen = name->len; args.filetype = name->type; - args.hashval = dp->i_mount->m_dirnameops->hashname (name); + args.hashval = dp->i_mount->m_dirnameops->hashname (name->name, + name->len); args.dp = dp; args.whichfork =XFS_DATA_FORK ; args.trans = tp;
libxfs/xfs_dir2_block.c +1 -4
@@ -1047,7 +1047,6 @@ xfs_dir2_sf_to_block(xfs_dir2_sf_hdr_t *sfp;/* shortform header */ __be16 *tagp;/* end of data entry */ xfs_trans_t *tp;/* transaction pointer */ -struct xfs_name name;struct xfs_ifork *ifp;trace_xfs_dir2_sf_to_block (args);
@@ -1205,10 +1204,8 @@ xfs_dir2_sf_to_block( tagp =xfs_dir3_data_entry_tag_p (mp, dep); *tagp =cpu_to_be16 ((char *)dep - (char *)hdr);xfs_dir2_data_log_entry (tp, bp, dep); - name.name = sfep->name; - name.len = sfep->namelen; blp[2 + i].hashval =cpu_to_be32 (mp->m_dirnameops-> -hashname (&name)); +hashname (sfep->name, sfep->namelen)); blp[2 + i].address =cpu_to_be32 (xfs_dir2_byte_to_dataptr (mp, (char *)dep - (char *)hdr)); offset = (int )((char *)(tagp +1 ) - (char *)hdr);
libxfs/xfs_dir2_data.c +2 -4
@@ -46,7 +46,6 @@ __xfs_dir3_data_check(xfs_mount_t *mp;/* filesystem mount point */ char *p;/* current data position */ int stale;/* count of stale leaves */ -struct xfs_name name; mp = bp->b_target->bt_mount; hdr = bp->b_addr;
@@ -142,9 +141,8 @@ __xfs_dir3_data_check( addr =xfs_dir2_db_off_to_dataptr (mp, mp->m_dirdatablk, (xfs_dir2_data_aoff_t) ((char *)dep - (char *)hdr)); - name.name = dep->name; - name.len = dep->namelen; - hash = mp->m_dirnameops->hashname (&name); + hash = mp->m_dirnameops-> +hashname (dep->name, dep->namelen);for (i =0 ; i <be32_to_cpu (btp->count); i++) {if (be32_to_cpu (lep[i].address) == addr &&be32_to_cpu (lep[i].hashval) == hash)
repair/phase6.c +1 -1
@@ -195,7 +195,7 @@ dir_hash_add( dup =0 ;if (!junk) { - hash = mp->m_dirnameops->hashname (&xname); + hash = mp->m_dirnameops->hashname (name, namelen); byhash =DIR_HASH_FUNC (hashtab, hash);/*
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 5/13 libxfs: add a superblock feature bit to indicate UTF-8 support.
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
When UTF-8 support is enabled, the xfs_dir_ci_inode_operations must be installed. Add xfs_sb_version_hasci(), which tests both the borgbit and the utf8bit, and returns true if at least one of them is set. Replace calls to xfs_sb_version_hasasciici() as needed.
--- include/xfs_fs.h | 2 +- include/xfs_sb.h | 25 ++++++++++++++++++++++++- 2 files changed, 25 insertions(+), 2 deletions(-)
include/xfs_fs.h +1 -1
@@ -239,7 +239,7 @@ typedef struct xfs_fsop_resblks {#define XFS_FSOP_GEOM_FLAGS_V5SB 0x8000/* version 5 superblock */ #define XFS_FSOP_GEOM_FLAGS_FTYPE 0x10000/* inode directory types */ #define XFS_FSOP_GEOM_FLAGS_FINOBT 0x20000/* free inode btree */ - +#define XFS_FSOP_GEOM_FLAGS_UTF8 0x40000/* utf8 filenames */ /* * Minimum and maximum sizes need for growth checks.
include/xfs_sb.h +24 -1
@@ -82,6 +82,8 @@ struct xfs_trans;#define XFS_SB_VERSION2_RESERVED4BIT 0x00000004#define XFS_SB_VERSION2_ATTR2BIT 0x00000008/* Inline attr rework */ #define XFS_SB_VERSION2_PARENTBIT 0x00000010/* parent pointers */ +#define XFS_SB_VERSION2_PARENTBIT 0x00000010/* parent pointers */ +#define XFS_SB_VERSION2_UTF8BIT 0x00000020/* utf8 names */ #define XFS_SB_VERSION2_PROJID32BIT 0x00000080/* 32 bit project id */ #define XFS_SB_VERSION2_CRCBIT 0x00000100/* metadata CRCs */ #define XFS_SB_VERSION2_FTYPE 0x00000200/* inode type in dir */
@@ -89,6 +91,7 @@ struct xfs_trans;#define XFS_SB_VERSION2_OKREALFBITS \ (XFS_SB_VERSION2_LAZYSBCOUNTBIT | \ XFS_SB_VERSION2_ATTR2BIT | \ + XFS_SB_VERSION2_UTF8BIT | \ XFS_SB_VERSION2_PROJID32BIT | \ XFS_SB_VERSION2_FTYPE)#define XFS_SB_VERSION2_OKSASHFBITS \
@@ -600,8 +603,10 @@ xfs_sb_has_ro_compat_feature( }#define XFS_SB_FEAT_INCOMPAT_FTYPE (1 << 0)/* filetype in dirent */ +#define XFS_SB_FEAT_INCOMPAT_UTF8 (1 << 1)/* utf-8 name support */ #define XFS_SB_FEAT_INCOMPAT_ALL \ - (XFS_SB_FEAT_INCOMPAT_FTYPE) + (XFS_SB_FEAT_INCOMPAT_FTYPE | \ + XFS_SB_FEAT_INCOMPAT_UTF8)#define XFS_SB_FEAT_INCOMPAT_UNKNOWN ~XFS_SB_FEAT_INCOMPAT_ALL static inline bool
@@ -649,6 +654,24 @@ static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp) (sbp->sb_features_ro_compat &XFS_SB_FEAT_RO_COMPAT_FINOBT ); } +static inline int xfs_sb_version_hasutf8 (xfs_sb_t *sbp) +{ +return (XFS_SB_VERSION_NUM (sbp) ==XFS_SB_VERSION_5 && +xfs_sb_has_incompat_feature (sbp,XFS_SB_FEAT_INCOMPAT_UTF8 )) || + (xfs_sb_version_hasmorebits (sbp) && + (sbp->sb_features2 &XFS_SB_VERSION2_UTF8BIT )); +} + +/* +* Special case: there are a number of places where we need to test +* both the borgbit and the utf8bit, and take the same action if +* either of those is set. +*/ +static inline int xfs_sb_version_hasci (xfs_sb_t *sbp) +{ +return xfs_sb_version_hasasciici (sbp) ||xfs_sb_version_hasutf8 (sbp); +} +/* * end of superblock version macros */
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 6/13 xfsprogs: add unicode character database files
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Add files from the Unicode Character Database, version 7.0.0, to the source. A helper program that generates a trie used for normalization from these files is part of a separate commit.
---
[v2: removed large unicode files. download them as below. -bpm]
cd support/ucd-7.0.0
wget http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt
wget http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt
wget http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt
wget http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt
wget http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt
wget http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt
wget http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt
--- support/ucd-7.0.0/README | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 support/ucd-7.0.0/README
support/ucd-7.0.0/README new file +33 -0
@@ -0,0 +1,33 @@ +The files in this directory are part of the Unicode Character Database +for version 7.0.0 of the Unicode standard. + +The full set of files can be found here: + + http://www.unicode.org/Public/7.0.0/ucd/ + +The latest released version of the UCD can be found here: + + http://www.unicode.org/Public/UCD/latest/ + +The files in this directory are identical, except that they have been +renamed with a suffix indicating the unicode version. + +Individual source links: + + http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt + http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt + http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt + http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt + http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt + http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt + http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt + +md5sums + + 9a92b2bfe56c6719def926bab524fefd CaseFolding-7.0.0.txt + 07b8b1027eb824cf0835314e94f23d2e DerivedAge-7.0.0.txt + 90c3340b16821e2f2153acdbe6fc6180 DerivedCombiningClass-7.0.0.txt + c41c0601f808116f623de47110ed4f93 DerivedCoreProperties-7.0.0.txt + 522720ddfc150d8e63a2518634829bce NormalizationCorrections-7.0.0.txt + 1f35175eba4a2ad795db489f789ae352 NormalizationTest-7.0.0.txt + c8355655731d75e6a3de8c20d7e601ba UnicodeData-7.0.0.txt
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 7/13 libxfs: add trie generator and supporting code for UTF-8.
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
mkutf8data.c is the source for a program that generates utf8data.h, which contains the trie that utf8norm.c uses. The trie is generated from the Unicode 7.0.0 data files. The format of the utf8data[] table is described in utf8norm.c.
Supporting functions for UTF-8 normalization are in utf8norm.c with the header utf8norm.h. Two normalization forms are supported: nfkdi and nfkdicf.
nfkdi:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
nfkdicf:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
- Apply a full casefold (C + F).
For the purposes of the code, a string is valid UTF-8 if:
- The values encoded are 0x1..0x10FFFF.
- The surrogate codepoints 0xD800..0xDFFFF are not encoded.
- The shortest possible encoding is used for all values.
The supporting functions work on null-terminated strings (utf8 prefix) and on length-limited strings (utf8n prefix).
--- include/utf8norm.h | 111 ++ libxfs/utf8norm.c | 628 ++++++++++ support/mkutf8data.c | 3232 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 3971 insertions(+) create mode 100644 include/utf8norm.h create mode 100644 libxfs/utf8norm.c create mode 100644 support/mkutf8data.c
include/utf8norm.h new file +111 -0
@@ -0,0 +1,111 @@ +/* +* Copyright (c) 2014 SGI. +* All rights reserved. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public License as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it would be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write the Free Software Foundation, +* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +*/ + +#ifndef UTF8NORM_H +#define UTF8NORM_H + +/* An opaque type used to determine the normalization in use. */ +typedef const struct utf8data *utf8data_t ; + +/* Encoding a unicode version number as a single unsigned int. */ +#define UNICODE_MAJ_SHIFT (16) +#define UNICODE_MIN_SHIFT (8) + +#define UNICODE_AGE (MAJ ,MIN ,REV ) \ + (((unsigned int)(MAJ) << UNICODE_MAJ_SHIFT) | \ + ((unsigned int)(MIN) << UNICODE_MIN_SHIFT) | \ + ((unsigned int)(REV))) + +/* Highest unicode version supported by the data tables. */ +extern const unsigned int utf8version; + +/* +* Look for the correct utf8data_t for a unicode version. +* Returns NULL if the version requested is too new. +* +* Two normalization forms are supported: nfkdi and nfkdicf. +* +* nfkdi: +* - Apply unicode normalization form NFKD. +* - Remove any Default_Ignorable_Code_Point. +* +* nfkdicf: +* - Apply unicode normalization form NFKD. +* - Remove any Default_Ignorable_Code_Point. +* - Apply a full casefold (C + F). +*/ +extern utf8data_t utf8nfkdi (unsigned int ); +extern utf8data_t utf8nfkdicf (unsigned int ); + +/* +* Determine the maximum age of any unicode character in the string. +* Returns 0 if only unassigned code points are present. +* Returns -1 if the input is not valid UTF-8. +*/ +extern int utf8agemax (utf8data_t ,const char *); +extern int utf8nagemax (utf8data_t ,const char *,size_t ); + +/* +* Determine the minimum age of any unicode character in the string. +* Returns 0 if any unassigned code points are present. +* Returns -1 if the input is not valid UTF-8. +*/ +extern int utf8agemin (utf8data_t ,const char *); +extern int utf8nagemin (utf8data_t ,const char *,size_t ); + +/* +* Determine the length of the normalized from of the string, +* excluding any terminating NULL byte. +* Returns 0 if only ignorable code points are present. +* Returns -1 if the input is not valid UTF-8. +*/ +extern ssize_t utf8len (utf8data_t ,const char *); +extern ssize_t utf8nlen (utf8data_t ,const char *,size_t ); + +/* +* Cursor structure used by the normalizer. +*/ +struct utf8cursor { +utf8data_t data; +const char *s; +const char *p; +const char *ss; +const char *sp; +unsigned int len; +unsigned int slen; +short int ccc; +short int nccc; +}; + +/* +* Initialize a utf8cursor to normalize a string. +* Returns 0 on success. +* Returns -1 on failure. +*/ +extern int utf8cursor (struct utf8cursor *,utf8data_t ,const char *); +extern int utf8ncursor (struct utf8cursor *,utf8data_t ,const char *,size_t ); + +/* +* Get the next byte in the normalization. +* Returns a value > 0 && < 256 on success. +* Returns 0 when the end of the normalization is reached. +* Returns -1 if the string being normalized is not valid UTF-8. +*/ +extern int utf8byte (struct utf8cursor *); + +#endif /* UTF8NORM_H */
libxfs/utf8norm.c new file +628 -0
628 lines not on this page · raw message
support/mkutf8data.c new file +3232 -0
2,261 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Part of the diff is not shown here. View the raw message.
PATCH 8/13 libxfs: add xfs_nameops for utf8 and utf8+casefold.
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
The xfs_utf8_nameops use the nfkdi normalization when comparing filenames, and are installed if the utf8bit is set in the super block.
The xfs_utf8_ci_nameops use the nfkdicf normalization when comparing filenames, and are installed if both the utf8bit and the borgbit are set in the superblock.
Normalized filenames are not stored on disk. Normalization will fail if a filename is not valid UTF-8, in which case the filename is treated as an opaque blob.
Changes:
Type conversion to "(const char *)" added to utf8ncursor() and utf8nlen() calls.
--- Makefile | 2 +- include/libxfs.h | 1 + include/xfs_utf8.h | 25 ++++++ libxfs/Makefile | 4 +- libxfs/xfs_dir2.c | 15 +++- libxfs/xfs_utf8.c | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++ support/Makefile | 24 ++++++ 7 files changed, 303 insertions(+), 6 deletions(-) create mode 100644 include/xfs_utf8.h create mode 100644 libxfs/xfs_utf8.c create mode 100644 support/Makefile
Makefile +1 -1
@@ -40,7 +40,7 @@ LDIRDIRT = $(SRCDIR) LDIRT += $(SRCTAR) endif -LIB_SUBDIRS = libxfs libxlog libxcmd libhandle libdisk +LIB_SUBDIRS = support libxfs libxlog libxcmd libhandle libdisk TOOL_SUBDIRS = copy db estimate fsck fsr growfs io logprint mkfs quota \ mdrestore repair rtcp m4 man doc po debian
include/libxfs.h +1 -0
@@ -59,6 +59,7 @@#include <xfs/xfs_btree_trace.h> #include <xfs/xfs_bmap.h> #include <xfs/xfs_trace.h> +#include <xfs_utf8.h> #ifndef ARRAY_SIZE
include/xfs_utf8.h new file +25 -0
@@ -0,0 +1,25 @@ +/* +* Copyright (c) 2014 SGI. +* All rights reserved. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public License as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it would be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write the Free Software Foundation, +* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +*/ + +#ifndef XFS_UTF8_H +#define XFS_UTF8_H + +extern struct xfs_nameops xfs_utf8_nameops; +extern struct xfs_nameops xfs_utf8_ci_nameops; + +#endif /* XFS_UTF8_H */
libxfs/Makefile +3 -1
@@ -14,6 +14,7 @@ HFILES = xfs.h init.h xfs_dir2_priv.h crc32defs.h crc32table.h CFILES = cache.c \ crc32.c \ init.c kmem.c logitem.c radix-tree.c rdwr.c trans.c util.c \ + utf8norm.c \ xfs_alloc.c \ xfs_alloc_btree.c \ xfs_attr.c \
@@ -38,7 +39,8 @@ CFILES = cache.c \ xfs_rtbitmap.c \ xfs_sb.c \ xfs_symlink_remote.c \ - xfs_trans_resv.c + xfs_trans_resv.c \ + xfs_utf8.c CFILES += $(PKG_PLATFORM).c PCFILES = darwin.c freebsd.c irix.c linux.c
libxfs/xfs_dir2.c +11 -4
@@ -123,10 +123,17 @@ xfs_dir_mount( (uint )sizeof (xfs_da_node_entry_t); mp->m_dir_magicpct = (mp->m_dirblksize *37 ) /100 ; -if (xfs_sb_version_hasasciici (&mp->m_sb)) - mp->m_dirnameops = &xfs_ascii_ci_nameops; -else - mp->m_dirnameops = &xfs_default_nameops; +if (xfs_sb_version_hasutf8 (&mp->m_sb)) { +if (xfs_sb_version_hasasciici (&mp->m_sb)) + mp->m_dirnameops = &xfs_utf8_ci_nameops; +else + mp->m_dirnameops = &xfs_utf8_nameops; + }else { +if (xfs_sb_version_hasasciici (&mp->m_sb)) + mp->m_dirnameops = &xfs_ascii_ci_nameops; +else + mp->m_dirnameops = &xfs_default_nameops; + } }/*
libxfs/xfs_utf8.c new file +238 -0
238 lines not on this page · raw message
support/Makefile new file +24 -0
@@ -0,0 +1,24 @@ +# +# Copyright (c) 2014 SGI. All Rights Reserved. +# + +TOPDIR = .. +include $(TOPDIR)/include/builddefs + +default = ../include/utf8data.h + +../include/utf8data.h: mkutf8data.c + cc -o mkutf8data mkutf8data.c + cd ucd-7.0.0 ; ../mkutf8data + mv ucd-7.0.0/utf8data.h ../include + +default clean: + rm -f mkutf8data ../include/utf8data.h + +default install: + +default install-dev: + +default install-qa: + +-include .ltdep
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 9/13 libxfs: apply utf-8 normalization rules to user extended attribute names
In reply to Ben MyersFrom: Olaf Weber <[email protected]>
Apply the same rules for UTF-8 normalization to the names of user-defined extended attributes. System attributes are excluded because they are not user-visible in the first place, and the kernel is expected to know what it is doing when naming them.
--- libxfs/xfs_attr.c | 49 +++++++++++++++++++++++++++++++++++++++++-------- libxfs/xfs_attr_leaf.c | 11 +++++++++-- libxfs/xfs_utf8.c | 7 +++++++ 3 files changed, 57 insertions(+), 10 deletions(-)
libxfs/xfs_attr.c +41 -8
@@ -88,8 +88,9 @@ xfs_attr_get_int( int *valuelenp, int flags) { -xfs_da_args_t args; -int error; +xfs_da_args_t args; +struct xfs_mount *mp = ip->i_mount; +int error;if (!xfs_inode_hasattr (ip))return ENOATTR ;
@@ -103,9 +104,12 @@ xfs_attr_get_int( args.value = value; args.valuelen = *valuelenp; args.flags = flags; - args.hashval =xfs_da_hashname (args.name, args.namelen); args.dp = ip; args.whichfork =XFS_ATTR_FORK ; +if (!xfs_sb_version_hasutf8 (&mp->m_sb)) + args.hashval =xfs_da_hashname (args.name, args.namelen); +else if ((error = mp->m_dirnameops->normhash (&args)) !=0 ) +return error;/* * Decide on what work routines to call based on the inode size.
@@ -118,6 +122,9 @@ xfs_attr_get_int( error =xfs_attr_node_get (&args); } +if (args.norm) +kmem_free ((void *)args.norm); +/* * Return the number of bytes in the value to the caller. */
@@ -239,12 +246,15 @@ xfs_attr_set_int( args.value = value; args.valuelen = valuelen; args.flags = flags; - args.hashval =xfs_da_hashname (args.name, args.namelen); args.dp = dp; args.firstblock = &firstblock; args.flist = &flist; args.whichfork =XFS_ATTR_FORK ; args.op_flags =XFS_DA_OP_ADDNAME |XFS_DA_OP_OKNOENT ; +if (!xfs_sb_version_hasutf8 (&mp->m_sb)) + args.hashval =xfs_da_hashname (args.name, args.namelen); +else if ((error = mp->m_dirnameops->normhash (&args)) !=0 ) +return error;/* Size is now blocks for attribute data */ args.total =xfs_attr_calc_size (dp, name->len, valuelen, &local);
@@ -276,6 +286,8 @@ xfs_attr_set_int( error =xfs_trans_reserve (args.trans, &tres, args.total,0 );if (error) {xfs_trans_cancel (args.trans,0 ); +if (args.norm) +kmem_free ((void *)args.norm);return (error); }xfs_ilock (dp,XFS_ILOCK_EXCL );
@@ -286,6 +298,8 @@ xfs_attr_set_int(if (error) {xfs_iunlock (dp,XFS_ILOCK_EXCL );xfs_trans_cancel (args.trans,XFS_TRANS_RELEASE_LOG_RES ); +if (args.norm) +kmem_free ((void *)args.norm);return (error); }
@@ -333,7 +347,8 @@ xfs_attr_set_int( err2 =xfs_trans_commit (args.trans,XFS_TRANS_RELEASE_LOG_RES );xfs_iunlock (dp,XFS_ILOCK_EXCL ); - +if (args.norm) +kmem_free ((void *)args.norm);return (error ==0 ? err2 : error); }
@@ -398,6 +413,8 @@ xfs_attr_set_int(xfs_trans_log_inode (args.trans, dp,XFS_ILOG_CORE ); error =xfs_trans_commit (args.trans,XFS_TRANS_RELEASE_LOG_RES );xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free ((void *)args.norm);return (error);
@@ -406,6 +423,9 @@ out:xfs_trans_cancel (args.trans,XFS_TRANS_RELEASE_LOG_RES |XFS_TRANS_ABORT );xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free ((void *)args.norm); +return (error); }
@@ -452,12 +472,15 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags) args.name = name->name; args.namelen = name->len; args.flags = flags; - args.hashval =xfs_da_hashname (args.name, args.namelen); args.dp = dp; args.firstblock = &firstblock; args.flist = &flist; args.total =0 ; args.whichfork =XFS_ATTR_FORK ; +if (!xfs_sb_version_hasutf8 (&mp->m_sb)) + args.hashval =xfs_da_hashname (args.name, args.namelen); +else if ((error = mp->m_dirnameops->normhash (&args)) !=0 ) +return error;/* * we have no control over the attribute names that userspace passes us
@@ -470,8 +493,11 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags)* Attach the dquots to the inode. */ error =xfs_qm_dqattach (dp,0 ); -if (error) -return error; +if (error) { +if (args.norm) +kmem_free ((void *)args.norm); +return error; + }/* * Start our first transaction of the day.
@@ -497,6 +523,8 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags) XFS_ATTRRM_SPACE_RES(mp), 0);if (error) {xfs_trans_cancel (args.trans,0 ); +if (args.norm) +kmem_free ((void *)args.norm);return (error); }
@@ -546,6 +574,8 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags)xfs_trans_log_inode (args.trans, dp,XFS_ILOG_CORE ); error =xfs_trans_commit (args.trans,XFS_TRANS_RELEASE_LOG_RES );xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free ((void *)args.norm);return (error);
@@ -554,6 +584,9 @@ out:xfs_trans_cancel (args.trans,XFS_TRANS_RELEASE_LOG_RES |XFS_TRANS_ABORT );xfs_iunlock (dp,XFS_ILOCK_EXCL ); +if (args.norm) +kmem_free ((void *)args.norm); +return (error); }
libxfs/xfs_attr_leaf.c +9 -2
@@ -634,6 +634,7 @@ int xfs_attr_shortform_to_leaf(xfs_da_args_t *args) {xfs_inode_t *dp; +struct xfs_mount *mp;xfs_attr_shortform_t *sf;xfs_attr_sf_entry_t *sfe;xfs_da_args_t nargs;
@@ -646,6 +647,7 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args)trace_xfs_attr_sf_to_leaf (args); dp = args->dp; + mp = dp->i_mount; ifp = dp->i_afp; sf = (xfs_attr_shortform_t *)ifp->if_u1.if_data; size =be16_to_cpu (sf->hdr.totsize);
@@ -698,13 +700,18 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args) nargs.namelen = sfe->namelen; nargs.value = &sfe->nameval[nargs.namelen]; nargs.valuelen = sfe->valuelen; - nargs.hashval =xfs_da_hashname (sfe->nameval, - sfe->namelen); nargs.flags =XFS_ATTR_NSP_ONDISK_TO_ARGS (sfe->flags); +if (!xfs_sb_version_hasutf8 (&mp->m_sb)) + nargs.hashval =xfs_da_hashname (sfe->nameval, + sfe->namelen); +else if ((error = mp->m_dirnameops->normhash (&nargs)) !=0 ) + goto out; error =xfs_attr3_leaf_lookup_int (bp, &nargs);/* set a->index */ ASSERT (error ==ENOATTR ); error =xfs_attr3_leaf_add (bp, &nargs);ASSERT (error !=ENOSPC ); +if (nargs.norm) +kmem_free ((void *)nargs.norm);if (error) goto out; sfe =XFS_ATTR_SF_NEXTENTRY (sfe);
libxfs/xfs_utf8.c +7 -0
@@ -31,6 +31,7 @@#include "xfs_inode_fork.h" #include "xfs_bmap.h" #include "xfs_dir2.h" +#include "xfs_attr_leaf.h" #include "xfs_trace.h" #include "xfs_utf8.h" #include "utf8norm.h"
@@ -72,6 +73,9 @@ xfs_utf8_normhash(ssize_t normlen;int c; +/* Don't normalize system attribute names. */ +if (args->flags & (ATTR_ROOT |ATTR_SECURE )) + goto blob; nfkdi =utf8nfkdi (utf8version);/* Failure to normalize is treated as a blob. */ if ((normlen = utf8nlen(nfkdi, (const char *)args->name,
@@ -173,6 +177,9 @@ xfs_utf8_ci_normhash(ssize_t normlen;int c; +/* Don't normalize system attribute names. */ +if (args->flags & (ATTR_ROOT |ATTR_SECURE )) + goto blob; nfkdicf =utf8nfkdicf (utf8version);/* Failure to normalize is treated as a blob. */ if ((normlen = utf8nlen(nfkdicf, (const char *)args->name,
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 10/13 xfsprogs: add utf8 support to growfs
In reply to Ben MyersFrom: Mark Tinguely <[email protected]>
Add utf-8 to xfs_growfs and xfs_info.
--- growfs/xfs_growfs.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
growfs/xfs_growfs.c +8 -5
@@ -57,7 +57,8 @@ report_info( int crcs_enabled, int cimode, int ftype_enabled, - int finobt_enabled) + int finobt_enabled, + int utf8) { printf(_("meta-data=%-22s isize=%-6u agcount=%u, agsize=%u blks\n"
@@ -65,7 +66,7 @@ report_info(" =%-22s crc=%-8u finobt=%u\n" "data =%-22s bsize=%-6u blocks=%llu, imaxpct=%u\n" " =%-22s sunit=%-6u swidth=%u blks\n" -"naming =version %-14u bsize=%-6u ascii-ci=%d ftype=%d\n" +"naming =version %-14u bsize=%-6u ascii-ci=%d ftype=%d utf8=%d\n" "log =%-22s bsize=%-6u blocks=%u, version=%u\n" " =%-22s sectsz=%-5u sunit=%u blks, lazy-count=%u\n" "realtime =%-22s extsz=%-6u blocks=%llu, rtextents=%llu\n"),
@@ -76,7 +77,7 @@ report_info("" , geo.blocksize, (unsigned long long )geo.datablocks, geo.imaxpct,"" , geo.sunit, geo.swidth, - dirversion, geo.dirblocksize, cimode, ftype_enabled, + dirversion, geo.dirblocksize, cimode, ftype_enabled, utf8, isint ?_ ("internal" ) : logname ? logname :_ ("external" ), geo.blocksize, geo.logblocks, logversion, "", geo.logsectsize, geo.logsunit / geo.blocksize, lazycount,
@@ -114,6 +115,7 @@ main(int argc, char **argv)long long rsize;/* new rt size in fs blocks */ int ci;/* ASCII case-insensitive fs */ int lazycount;/* lazy superblock counters */ +int utf8;/* Unicode chars supported */ int xflag;/* -x flag */ char *fname;/* mount point name */ char *datadev;/* data device name */
@@ -247,11 +249,12 @@ main(int argc, char **argv) crcs_enabled = geo.flags &XFS_FSOP_GEOM_FLAGS_V5SB ?1 :0 ; ftype_enabled = geo.flags &XFS_FSOP_GEOM_FLAGS_FTYPE ?1 :0 ; finobt_enabled = geo.flags &XFS_FSOP_GEOM_FLAGS_FINOBT ?1 :0 ; + utf8 = geo.flags &XFS_FSOP_GEOM_FLAGS_UTF8 ?1 :0 ;if (nflag) {report_info (geo, datadev, isint, logdev, rtdev, lazycount, dirversion, logversion, attrversion, projid32bit, crcs_enabled, ci, - ftype_enabled, finobt_enabled); + ftype_enabled, finobt_enabled, utf8);exit (0 ); }
@@ -289,7 +292,7 @@ main(int argc, char **argv)report_info (geo, datadev, isint, logdev, rtdev, lazycount, dirversion, logversion, attrversion, projid32bit, crcs_enabled, ci, ftype_enabled, - finobt_enabled); + finobt_enabled, utf8); ddsize = xi.dsize; dlsize = ( xi.logBBsize? xi.logBBsize :
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 11/13 xfsprogs: add utf8 support to mkfs.xfs
In reply to Ben MyersFrom: Mark Tinguely <[email protected]>
Set the utf-8 feature bit.
--- man/man8/mkfs.xfs.8 | 9 ++++++++- mkfs/xfs_mkfs.c | 27 ++++++++++++++++++++++----- mkfs/xfs_mkfs.h | 3 ++- 3 files changed, 32 insertions(+), 7 deletions(-)
man/man8/mkfs.xfs.8 +8 -1
@@ -558,7 +558,7 @@ any power of 2 size from the filesystem block size up to 65536. .IP The .B version=ci -option enables ASCII only case-insensitive filename lookup and version +option enables ASCII or UTF-8 case-insensitive filename lookup and version 2 directories. Filenames are case-preserving, that is, the names are stored in directories using the case they were created with. .IP
@@ -582,6 +582,13 @@ When CRCs are enabled via the ftype functionality is always enabled. This feature can not be turned off for such filesystem configurations. .IP +.TP +.BI utf8[= value ] +This is used to enable the UTF-8 character set support. The +.I value +is either 0 or 1, with 1 signifying that UTF-8 character support is to be +enabled. If the value is omitted, 1 is assumed. +.IP .RE .TP .BI \-p " protofile"
mkfs/xfs_mkfs.c +22 -5
@@ -149,6 +149,8 @@ char *nopts[] = { "version",#define N_FTYPE 3 "ftype", +#define N_UTF8 4 +"utf8" , NULL, };
@@ -958,6 +960,7 @@ main(int nsflag;int nvflag;int nci; +int utf8;int Nflag;int discard =1 ;char *p;
@@ -1004,6 +1007,7 @@ main( logagno = logblocks = rtblocks = rtextblocks =0 ; Nflag = nlflag = nsflag = nvflag = nci =0 ; nftype = dirftype =0 ;/* inode type information in the dir */ + utf8 =0 ;/* utf-8 support */ dirblocklog = dirblocksize =0 ; dirversion =XFS_DFL_DIR_VERSION ; qflag =0 ;
@@ -1565,7 +1569,8 @@ _("cannot specify both crc and ftype\n"));if (nvflag)respec ('n' , nopts,N_VERSION );if (!strcasecmp (value,"ci" )) { - nci =1 ;/* ASCII CI mode */ +/* ASCII or UTF-8 CI mode */ + nci =1 ; }else { dirversion =atoi (value);if (dirversion !=2 )
@@ -1587,6 +1592,14 @@ _("cannot specify both crc and ftype\n")); } nftype =1 ;break ; +case N_UTF8 : +if (!value || *value =='\0' ) + value ="1" ; + c =atoi (value); +if (c <0 || c >1 ) +illegal (value,"n utf8" ); + utf8 = c; +break ;default :unknown ('n' , value); }
@@ -2460,7 +2473,8 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"),*/ sbp->sb_features2 =XFS_SB_VERSION2_MKFS (crcs_enabled, lazy_sb_counters, attrversion ==2 , !projid16bit,0 , - (!crcs_enabled && dirftype)); + (!crcs_enabled && dirftype), + (!crcs_enabled && utf8)); sbp->sb_versionnum = XFS_SB_VERSION_MKFS(crcs_enabled, iaflag, dsunit != 0, logversion == 2, attrversion == 1,
@@ -2534,6 +2548,9 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"),if (crcs_enabled) { sbp->sb_features_incompat =XFS_SB_FEAT_INCOMPAT_FTYPE ; dirftype =1 ; +/* turn on the utf-8 support */ +if (utf8) + sbp->sb_features_incompat |=XFS_SB_FEAT_INCOMPAT_UTF8 ; }if (!qflag || Nflag) {
@@ -2543,7 +2560,7 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n")," =%-22s crc=%-8u finobt=%u\n" "data =%-22s bsize=%-6u blocks=%llu, imaxpct=%u\n" " =%-22s sunit=%-6u swidth=%u blks\n" -"naming =version %-14u bsize=%-6u ascii-ci=%d ftype=%d\n" +"naming =version %-14u bsize=%-6u ascii-ci=%d ftype=%d utf8=%d\n" "log =%-22s bsize=%-6d blocks=%lld, version=%d\n" " =%-22s sectsz=%-5u sunit=%d blks, lazy-count=%d\n" "realtime =%-22s extsz=%-6d blocks=%lld, rtextents=%lld\n"),
@@ -2552,7 +2569,7 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"),"" , crcs_enabled, finobt,"" , blocksize, (long long )dblocks, imaxpct,"" , dsunit, dswidth, - dirversion, dirblocksize, nci, dirftype, + dirversion, dirblocksize, nci, dirftype, utf8, logfile,1 << blocklog, (long long )logblocks, logversion,"" , lsectorsize, lsunit, lazy_sb_counters, rtfile, rtextblocks << blocklog,
@@ -3171,7 +3188,7 @@ usage( void ) sunit=value|su=num,sectlog=n|sectsize=num,\n\ lazy-count=0|1]\n\ /* label */ [-L label (maximum 12 characters)]\n\ -/* naming */ [-n log=n|size=num,version=2|ci,ftype=0|1]\n\ +/* naming */ [-n log=n|size=num,version=2|ci,ftype=0|1,utf8=0|1]\n\ /* no-op info only */ [-N]\n\ /* prototype file */ [-p fname]\n\ /* quiet */ [-q]\n\
mkfs/xfs_mkfs.h +2 -1
@@ -37,13 +37,14 @@ 0 ) : XFS_SB_VERSION_1 )#define XFS_SB_VERSION2_MKFS (crc, lazycount, attr2, projid32bit, parent, \ - ftype) (\ + ftype, utf8) (\ ((lazycount) ? XFS_SB_VERSION2_LAZYSBCOUNTBIT : 0) | \ ((attr2) ? XFS_SB_VERSION2_ATTR2BIT : 0) | \ ((projid32bit) ? XFS_SB_VERSION2_PROJID32BIT : 0) | \ ((parent) ? XFS_SB_VERSION2_PARENTBIT : 0) | \ ((crc) ? XFS_SB_VERSION2_CRCBIT : 0) | \ ((ftype) ? XFS_SB_VERSION2_FTYPE : 0) | \ + ((utf8) ? XFS_SB_VERSION2_UTF8BIT : 0) | \ 0 )#define XFS_DFL_BLOCKSIZE_LOG 12/* 4096 byte blocks */
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 12/13 xfsprogs: add utf8 support to xfs_repair
In reply to Ben MyersFrom: Mark Tinguely <[email protected]>
Fix the duplicate filename detection to use the utf-8 normalization routines.
--- repair/phase6.c | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-)
repair/phase6.c +25 -10
@@ -176,13 +176,15 @@ dir_hash_add(unsigned char *name, __uint8_t ftype) { -xfs_dahash_t hash =0 ; int byaddr;int byhash =0 ;dir_hash_ent_t *p;int dup;short junk;struct xfs_name xname; +xfs_da_args_t args; + +memset (&args,0 ,sizeof (xfs_da_args_t));ASSERT (!hashtab->names_duped);
@@ -195,19 +197,30 @@ dir_hash_add( dup =0 ;if (!junk) { - hash = mp->m_dirnameops->hashname (name, namelen); - byhash =DIR_HASH_FUNC (hashtab, hash); +int error; + + args.name = name; + args.namelen = namelen; + args.inumber = inum; + args.whichfork =XFS_DATA_FORK ; + + error = mp->m_dirnameops->normhash (&args); +if (error) +do_error (_ ("normalize has failed %d)\n" ), error); + + byhash =DIR_HASH_FUNC (hashtab, args.hashval);/* * search hash bucket for existing name. */ for (p = hashtab->byhash[byhash]; p; p = p->nextbyhash) { -if (p->hashval == hash && p->name.len == namelen) { -if (memcmp (p->name.name, name, namelen) ==0 ) { - dup =1 ; - junk =1 ; -break ; - } +if (p->hashval == args.hashval && + mp->m_dirnameops->compname (&args, p->name.name, + p->name.len) != +XFS_CMP_DIFFERENT ) { + dup =1 ; + junk =1 ; +break ; } } }
@@ -226,7 +239,7 @@ dir_hash_add( hashtab->last = p;if (!(p->junkit = junk)) { - p->hashval = hash; + p->hashval = args.hashval; p->nextbyhash = hashtab->byhash[byhash]; hashtab->byhash[byhash] = p; }
@@ -235,6 +248,8 @@ dir_hash_add( p->seen =0 ; p->name = xname; +if (args.norm) +kmem_free ((void *) args.norm);return !dup; }
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 13/13 xfsprogs: add a preliminary test for utf8 support
In reply to Ben MyersFrom: Ben Myers <[email protected]>
Here's a preliminary test for utf8 support in xfs. It is based on Olaf's code that does some testing in the trie generator. Here too we are using the NormalizationTest.txt file from the unicode distribution. We check that the normalization in libxfs is working and then run checks on a filesystem. Note that there are some 'blacklisted' unichars which normalize to reserved characters.
FIXME:
For convenience of build this patch is against xfsprogs access to libxfs. Handling of ignorables and case fold is also not implemented here.
--- Makefile | 2 +- chkutf8data/Makefile | 21 +++ chkutf8data/chkutf8data.c | 430 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 452 insertions(+), 1 deletion(-) create mode 100644 chkutf8data/Makefile create mode 100644 chkutf8data/chkutf8data.c
Makefile +1 -1
@@ -42,7 +42,7 @@ endif LIB_SUBDIRS = support libxfs libxlog libxcmd libhandle libdisk TOOL_SUBDIRS = copy db estimate fsck fsr growfs io logprint mkfs quota \ - mdrestore repair rtcp m4 man doc po debian + mdrestore repair rtcp m4 man doc po debian chkutf8data SUBDIRS = include $(LIB_SUBDIRS) $(TOOL_SUBDIRS)
chkutf8data/Makefile new file +21 -0
@@ -0,0 +1,21 @@ +# +# Copyright (c) 2014 SGI. All Rights Reserved. +# + +TOPDIR = .. +include $(TOPDIR)/include/builddefs + +LTCOMMAND = chkutf8data +CFILES = chkutf8data.c + +LLDLIBS = $(LIBXFS) +LTDEPENDENCIES = $(LIBXFS) +LLDFLAGS = -static + +default: depend $(LTCOMMAND) + +include $(BUILDRULES) + +install: default + +-include .ltdep
chkutf8data/chkutf8data.c new file +430 -0
430 lines not on this page · raw message
Signature · 6 lines
1.7.12.4 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Ben Myers
Hi,
I'm posting this RFC on Olaf's behalf, as he is busy with other projects.
Ok, but I'd prefer to have Olaf discuss the finer points rather than have to play chinese whispers through you. :/
Quoted 2 lines
First is a series of kernel patches, then a series of patches for xfsprogs, and then a test.
Seeing as this is something out of the blue (i.e. nobody has made a mention of this functionality in the past couple of years), I think we need to look at design and architecture first before spending any time commenting on the code.
Quoted 3 lines
Note that I have removed the unicode database files prior to posting due to their large size. There are instructions on how to download them in the relevant commit headers.
Which leads to an interesting issue: these files do not have cryptographically verifiable signatures. How can I trust them? I can't even access unicode.org via https, so I can't even be certain that I'm downloading from the site I think I'm downloading from....
Quoted 16 lines
Here are some notes of introduction from Olaf:
-----------------------------------------------------------------------------
Unicode/UTF-8 support for XFSSo we had a customer request proper unicode support...
Design notes.
XFS uses byte strings for filenames, so UTF-8 is the expected format for unicode filenames. This does raise the question what criteria a byte string must meet to be UTF-8. We settled on the following:
- Valid unicode code points are 0..0x10FFFF, except that
- The surrogates 0xD800..0xDFFF are not valid code points, and
- Valid UTF-8 must be a shortest encoding of a valid unicode code point.
Quoted 6 lines
In addition, U+0 (ASCII NUL, '\0') is used to terminate byte strings (and is itself not part of the string). Moreover strings may be length-limited in addition to being NUL-terminated (there is no such thing as an embedded NUL in a length-limited string).
Based on feedback on the earlier patches for unicode/UTF-8 support, we
References, please. I don't recall any series discussion on this topic since Barry posted the unicode-CI patches back in 2008, and I doubt anyone remembers the details of those discussions....
Quoted 4 lines
decided that a filename that does not match the above criteria should be treated as a binary blob, as opposed to being rejected. To stress: if any part of the string isn't valid UTF-8, then the entire string is treated as a binary blob. This matters once normalization is considered.
So we accept invalid unicode in filenames, but only after failing to parse them? Isn't this a potential vector for exploiting weaknesses in application filename handling? i.e. unprivileged user writes specially crafted invalid unicode filename to disk, setuid program tries to parse it, invalid sequence triggers a buffer overflow bug in setuid parser?
Quoted 7 lines
When comparing unicode strings for equality, normalization comes into play:
we must compare the normalized forms of strings, not just the raw sequences of bytes. There are a number of defined normalization forms for unicode. We decided on a variant of NFKD we call NFKDI. NFD was chosed over NFC, because calculating NFC requires calculating NFD first, followed by an additional step. NFKD was chosen over NFD because this makes filenames that ought to be equal compare as equal.
But are they really equal?
Choosing *compatibility* decomposition over *canonical* decomposition means that compound characters and formatting distinctions don't affect the hash. i.e. "of'fi'ce", "o'ffi'ce" and "office" all hash and compare as the same name, but then they get stored on disk unnormalised. So they are the "same" in memory, but very different on disk.
I note that the unicode spec says this for normalised forms (11.1):
"A normalized string is guaranteed to be stable; that is, once normalized, a string is normalized according to all future versions of Unicode."
So if we store normalised strings on disk, they are guaranteed to be compatible with all future versions of unicode and anything that goes to use them. So why wouldn't we store normalised forms on disk?
As another point to note and discuss, from the unicode standard:
"Normalization Forms KC and KD must not be blindly applied to arbitrary text. [...] It is best to think of these Normalization Forms as being like uppercase or lowercase mappings: useful in certain contexts for identifying core meanings, but also performing modifications to the text that may not always be appropriate."
I'd consider file names to be mostly "arbitrary text" - we currently treat them as opaque blobs and don't try to interpret them (apart from '/' delimiters) and so they can contain arbitrary text....
Quoted 8 lines
My favorite example is the ways
"office" can be spelled, when "fi" or "ffi" ligatures are used. NFKDI adds one more step of NFKD, in that it eliminates the code points that have theDefault_Ignorable_Code_Pointproperty from the comparison. These code points are as a rule invisible, but might (or might not) be pulled in when you copy/paste a string to be used as a filename. An example of these is U+00AD SOFT HYPHEN, a code point that only shows up if a word is split across lines.
This extension does not appear to be specified by the unicode standard - this seems like a dangerous thing to do when considering compatibility with future unicode standards - we are not in the business of extend-and-embrace here. Anyway, what happens if a user actually wants a filename with a Default_Ignorable_Code_Point character in it?
IMO, if cut-n-paste modifies the string being cut-n-pasted, then that's a bug in the cut-n-paste application. I'd much prefer we use a normalisation type that is defined by the standard than to invent a new one to work around problems that may not even exist.
Quoted 2 lines
If a filename is considered to be binary blob, comparison is based on a simple binary match. Normalization does not apply to any part of a blob.
See above: if we have unicode enabled, I think that we should reject invalid unicode in filenames at normalisation time.
Quoted 5 lines
The code uses ("leverages", in corp-speak) the existing infrastructure for case-insensitive filenames. Like the CI code, the name used to create a file is stored on disk, and returned in a lookup. When comparing filenames the normalized forms of the names being compared are generated on the fly from the non-normalized forms stored on disk.
Again, why not store normalised forms on disk and avoid the need to generate normalised forms for dirents being read from disk every time they must be compared?
Quoted 4 lines
If the borgbit (the bit enabling legacy ASCII-based CI) is set in the superblock, then case folding is added into the mix. This normalization form we call NFKDICF. It allows for the creation of case-insensitive filesystems with UTF-8 support.
Different languages have different case folding rules e.g. the upper case character might be the same, but the lower case character is different (or vice versa). Where are the language specific case folding tables being stored? And speaking of language support, how does this interact with the kernel NLS subsystem?
Quoted 8 lines
-----------------------------------------------------------------------------
Implementation notes.Strings are normalized using a trie that stores the relevant information. The trie itself is part of the XFS module, and about 250kB in size. The trie is not checked in: instead we add the source files from the Unicode Character Database and a program that creates the header containing the trie.
This is rather unappealing. Distros would have to take this code size penalty if they decide one user needs that support. The other millions of users pay that cost even if they don't want it. And then there's validation - how are we supposed to validate that a 250k binary blob is correct and free of issues on every compiler and architecture that the kernel is built on?
Quoted 11 lines
The key for a lookup in the trie is a UTF-8 sequence. Each valid UTF-8 sequence leads to a leaf. No invalid sequence does. This means that trie lookups can be used to validate UTF-8 sequences, which why there is no specialized code for the same purpose.
The trie contains information for the version of unicode in which each code point was defined. This matters because non-normalized strings are stored on disk, and newer versions of unicode may introduce new normalized forms. Ideally, the version of unicode used by the filesystem is stored in the filesystem.
Quoted 4 lines
The trie also accounts for corrections made in the past to normalizations. This has little value today, because any newly created filesystem would be using unicode version 7.0.0. It is included in order to show, not tell, that such corrections can be handled if they are added in future revisions.
And so back to the stability of normalised forms: if the normalised forms are stable and the trie encodes the version of codepoints, then the data in the leaves of the trie itself must be stable. i.e. even for future versions of the standards, all the leaves that are there now will be there in the future. What is valid unicode now will remain valid unicode.
And given that, why do we need to carry the trie around in the compiled kernel? We have a perfectly good mechanism for storing large chunks of long-term stable metadata that we can access easily:
in files.
IOWs, the trie is really a property of the filesystem, not the kernel or userspace tools. If we ever want to update to a new version of unicode, we can compile a new trie and have mkfs write that into new filesystems, and maybe add an xfs-reapir function that allows migration to a new trie on an existing filesystem. But if we carry it in the kernel then there will be interesting issues with iupgrade/downgrade compatibility with new tries. Better to prevent those simply by havingthe trie be owned by the filesystem, not the kernel.
Hence I think the trie should probably be stored on disk in the filesystem. It gets calculated and written by mkfs into file attached to the superblock, and the only code that needs to go into the kernel is the code needed to read it into memory and walk it.
That means we don't need 3,000 lines of nasty trie generation code in the kernel, we don't bloat the kernel unnecessarily with abinary blob, we don't need to build code with data from unverifiable sources directly into the kernel, we can support different versions of unicode easily, and so on.
Quoted 3 lines
The algorithm used to calculate the sequences of bytes for the normalized form of a UTF-8 string is tricky. The core is found in
utf8byte(), with an explanation in the preceeding comment.
Precisely my point - it's nasty, tricky code, and getting it wrong is a potential security vulnerability. Exactly how are we expected to review >3,000 lines of unicode/utf-8 minutae without having to become unicode encoding experts?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Dave Chinner
Quoted 3 lines · Ben Myers
Hi,
I'm posting this RFC on Olaf's behalf, as he is busy with other projects.
Ok, but I'd prefer to have Olaf discuss the finer points rather than have to play chinese whispers through you. :/
I am on this mailing list, and I am trying to follow along, but I do have other calls on my time.
Quoted 8 lines
Quoted 2 lines
First is a series of kernel patches, then a series of patches for xfsprogs, and then a test.
Seeing as this is something out of the blue (i.e. nobody has made a mention of this functionality in the past couple of years), I think we need to look at design and architecture first before spending any time commenting on the code.
Quoted 8 lines
Quoted 3 lines
Note that I have removed the unicode database files prior to posting due to their large size. There are instructions on how to download them in the relevant commit headers.
Which leads to an interesting issue: these files do not have cryptographically verifiable signatures. How can I trust them? I can't even access unicode.org via https, so I can't even be certain that I'm downloading from the site I think I'm downloading from....
As Ben noted, the reason to not include them in these emails is their size:
$ wc fs/xfs/support/ucd/*
1273 12288 68009 fs/xfs/support/ucd/CaseFolding-7.0.0.txt
1470 14166 98263 fs/xfs/support/ucd/DerivedAge-7.0.0.txt
2368 22320 145072 fs/xfs/support/ucd/DerivedCombiningClass-7.0.0.txt
10794 123871 899859 fs/xfs/support/ucd/DerivedCoreProperties-7.0.0.txt
50 318 2040 fs/xfs/support/ucd/NormalizationCorrections-7.0.0.txt
18635 332441 2457187 fs/xfs/support/ucd/NormalizationTest-7.0.0.txt
33 86 1364 fs/xfs/support/ucd/README
27268 120686 1509570 fs/xfs/support/ucd/UnicodeData-7.0.0.txt
61891 626176 5181364 totalAs for your remarks about cryptographic signatures, I'm not sure I see your point there. Just to be clear: the idea is to check the files in, as opposed to having to download them from unicode.org prior to compiling XFS.
Quoted 21 lines
Quoted 21 lines
Here are some notes of introduction from Olaf:
-----------------------------------------------------------------------------
Unicode/UTF-8 support for XFSSo we had a customer request proper unicode support...
Design notes.
XFS uses byte strings for filenames, so UTF-8 is the expected format for unicode filenames. This does raise the question what criteria a byte string must meet to be UTF-8. We settled on the following:
- Valid unicode code points are 0..0x10FFFF, except that
- The surrogates 0xD800..0xDFFF are not valid code points, and
- Valid UTF-8 must be a shortest encoding of a valid unicode code point.In addition, U+0 (ASCII NUL, '\0') is used to terminate byte strings (and is itself not part of the string). Moreover strings may be length-limited in addition to being NUL-terminated (there is no such thing as an embedded NUL in a length-limited string).
Quoted 5 lines
Quoted 1 line
Based on feedback on the earlier patches for unicode/UTF-8 support, we
References, please. I don't recall any series discussion on this topic since Barry posted the unicode-CI patches back in 2008, and I doubt anyone remembers the details of those discussions....
I looked up those discussions in the archives. For example, here's Christoph about rejecting filenames if they're not well-formed unicode.
http://marc.info/?l=linux-fsdevel&m=120876935526856&w=2
And Jamie Lokier making a similar point:
http://oss.sgi.com/archives/xfs/2008-04/msg01263.html
Quoted 5 lines
Quoted 4 lines
decided that a filename that does not match the above criteria should be treated as a binary blob, as opposed to being rejected. To stress: if any part of the string isn't valid UTF-8, then the entire string is treated as a binary blob. This matters once normalization is considered.
Quoted 7 lines
So we accept invalid unicode in filenames, but only after failing to parse them? Isn't this a potential vector for exploiting weaknesses in application filename handling? i.e. unprivileged user writes specially crafted invalid unicode filename to disk, setuid program tries to parse it, invalid sequence triggers a buffer overflow bug in setuid parser?
Yes, this means that userspace must be capable of handling filenames that are not well-formed UTF-8 and a whole slew of other edge cases. Same as today really.
Quoted 17 lines
Quoted 7 lines
When comparing unicode strings for equality, normalization comes into play:
we must compare the normalized forms of strings, not just the raw sequences of bytes. There are a number of defined normalization forms for unicode. We decided on a variant of NFKD we call NFKDI. NFD was chosed over NFC, because calculating NFC requires calculating NFD first, followed by an additional step. NFKD was chosen over NFD because this makes filenames that ought to be equal compare as equal.But are they really equal?
Choosing *compatibility* decomposition over *canonical* decomposition means that compound characters and formatting distinctions don't affect the hash. i.e. "of'fi'ce", "o'ffi'ce" and "office" all hash and compare as the same name, but then they get stored on disk unnormalised. So they are the "same" in memory, but very different on disk.
Quoted 6 lines
I note that the unicode spec says this for normalised forms (11.1):
"A normalized string is guaranteed to be stable; that is, once normalized, a string is normalized according to all future versions of Unicode."
Provided no unassigned codepoints are present in that string.
Quoted 4 lines
So if we store normalised strings on disk, they are guaranteed to be compatible with all future versions of unicode and anything that goes to use them. So why wouldn't we store normalised forms on disk?
Because, based what I read around the web, I expect a good deal of resistance to the idea that a filesystem will on a lookup of a file you just created return a name that is different-but-equivalent.
Think of it as the equivalent of being case-preserving for a case-insensitive filesystem.
An alternative would be to store each filename twice: both raw and normalized forms.
Quoted 8 lines
As another point to note and discuss, from the unicode standard:
"Normalization Forms KC and KD must not be blindly applied to arbitrary text. [...] It is best to think of these Normalization Forms as being like uppercase or lowercase mappings: useful in certain contexts for identifying core meanings, but also performing modifications to the text that may not always be appropriate."
Quoted 4 lines
I'd consider file names to be mostly "arbitrary text" - we currently treat them as opaque blobs and don't try to interpret them (apart from '/' delimiters) and so they can contain arbitrary text....
My reading of this part of the unicode standard is that applying a compatibility normalization results in strings that materially differ from the originals, and no full equivalent of the original can be reconstructed from the normalized form. This makes it improper for a word processor to normalize to NFKC or NFKD before saving a file.
For the same reason, it would not be proper to store the NFKD version of a filename on disk without some method to retrieve (an equivalent of) the original.
Quoted 9 lines
Quoted 8 lines
My favorite example is the ways
"office" can be spelled, when "fi" or "ffi" ligatures are used. NFKDI adds one more step of NFKD, in that it eliminates the code points that have theDefault_Ignorable_Code_Pointproperty from the comparison. These code points are as a rule invisible, but might (or might not) be pulled in when you copy/paste a string to be used as a filename. An example of these is U+00AD SOFT HYPHEN, a code point that only shows up if a word is split across lines.
Quoted 6 lines
This extension does not appear to be specified by the unicode standard - this seems like a dangerous thing to do when considering compatibility with future unicode standards - we are not in the business of extend-and-embrace here. Anyway, what happens if a user actually wants a filename with a
Default_Ignorable_Code_Pointcharacter in it?
Such a filename can be created, and since the raw form of the name is stored on disk, when the filename is read back the Default_Ignorable_Code_Point will still be there. It just doesn't count when comparing names for equality.
Quoted 11 lines
IMO, if cut-n-paste modifies the string being cut-n-pasted, then that's a bug in the cut-n-paste application. I'd much prefer we use a normalisation type that is defined by the standard than to invent a new one to work around problems that may not even exist.
Quoted 2 lines
If a filename is considered to be binary blob, comparison is based on a simple binary match. Normalization does not apply to any part of a blob.
See above: if we have unicode enabled, I think that we should reject invalid unicode in filenames at normalisation time.
That was my original intent, which I abandoned based on the emails linked to above.
Quoted 15 lines
Quoted 5 lines
The code uses ("leverages", in corp-speak) the existing infrastructure for case-insensitive filenames. Like the CI code, the name used to create a file is stored on disk, and returned in a lookup. When comparing filenames the normalized forms of the names being compared are generated on the fly from the non-normalized forms stored on disk.
Again, why not store normalised forms on disk and avoid the need to generate normalised forms for dirents being read from disk every time they must be compared?
Quoted 4 lines
If the borgbit (the bit enabling legacy ASCII-based CI) is set in the superblock, then case folding is added into the mix. This normalization form we call NFKDICF. It allows for the creation of case-insensitive filesystems with UTF-8 support.
Quoted 5 lines
Different languages have different case folding rules e.g. the upper case character might be the same, but the lower case character is different (or vice versa). Where are the language specific case folding tables being stored? And speaking of language support, how does this interact with the kernel NLS subsystem?
I use a full case fold as per CaseFolding.txt to obtain a result that is consistent and (in my opinion) good enough.
Since XFS has no nls mount options, there is no interaction with the NLS subsystem.
Quoted 9 lines
Quoted 8 lines
-----------------------------------------------------------------------------
Implementation notes.Strings are normalized using a trie that stores the relevant information. The trie itself is part of the XFS module, and about 250kB in size. The trie is not checked in: instead we add the source files from the Unicode Character Database and a program that creates the header containing the trie.
Quoted 6 lines
This is rather unappealing. Distros would have to take this code size penalty if they decide one user needs that support. The other millions of users pay that cost even if they don't want it. And then there's validation - how are we supposed to validate that a 250k binary blob is correct and free of issues on every compiler and architecture that the kernel is built on?
If your concern is that the generator might create bad blobs on some architectures, then there are ways around that: checksums, checking in a reference blob, or maybe something else.
As for size in general, looking at the NLS support I do not consider it to be excessively big (as in, it is a bit less than 2 times the size of the largest NLS module). Obviously opinions can differ on this.
Quoted 16 lines
Quoted 11 lines
The key for a lookup in the trie is a UTF-8 sequence. Each valid UTF-8 sequence leads to a leaf. No invalid sequence does. This means that trie lookups can be used to validate UTF-8 sequences, which why there is no specialized code for the same purpose.
The trie contains information for the version of unicode in which each code point was defined. This matters because non-normalized strings are stored on disk, and newer versions of unicode may introduce new normalized forms. Ideally, the version of unicode used by the filesystem is stored in the filesystem.
Quoted 4 lines
The trie also accounts for corrections made in the past to normalizations. This has little value today, because any newly created filesystem would be using unicode version 7.0.0. It is included in order to show, not tell, that such corrections can be handled if they are added in future revisions.
Quoted 6 lines
And so back to the stability of normalised forms: if the normalised forms are stable and the trie encodes the version of codepoints, then the data in the leaves of the trie itself must be stable. i.e. even for future versions of the standards, all the leaves that are there now will be there in the future. What is valid unicode now will remain valid unicode.
The set of valid unicode code points is known and stable: 0..0x10FFFF minus 0xD800..0xDFFF. However, the set of assigned code points grows with each revision of the unicode standard. Note that there is an explicit limitation on the stability of normalized strings: they are stable if, and only if, no unassigned codepoints are present in the string.
Quoted 20 lines
And given that, why do we need to carry the trie around in the compiled kernel? We have a perfectly good mechanism for storing large chunks of long-term stable metadata that we can access easily:
in files.IOWs, the trie is really a property of the filesystem, not the kernel or userspace tools. If we ever want to update to a new version of unicode, we can compile a new trie and have mkfs write that into new filesystems, and maybe add an xfs-reapir function that allows migration to a new trie on an existing filesystem. But if we carry it in the kernel then there will be interesting issues with iupgrade/downgrade compatibility with new tries. Better to prevent those simply by havingthe trie be owned by the filesystem, not the kernel.
Hence I think the trie should probably be stored on disk in the filesystem. It gets calculated and written by mkfs into file attached to the superblock, and the only code that needs to go into the kernel is the code needed to read it into memory and walk it.
Quoted 5 lines
That means we don't need 3,000 lines of nasty trie generation code in the kernel, we don't bloat the kernel unnecessarily with abinary blob, we don't need to build code with data from unverifiable sources directly into the kernel, we can support different versions of unicode easily, and so on.
Storing the trie in the filesystem is certainly an option, as is making XFS UTF-8 support a config option.
Quoted 8 lines
Quoted 3 lines
The algorithm used to calculate the sequences of bytes for the normalized form of a UTF-8 string is tricky. The core is found in
utf8byte(), with an explanation in the preceeding comment.Precisely my point - it's nasty, tricky code, and getting it wrong is a potential security vulnerability. Exactly how are we expected to review >3,000 lines of unicode/utf-8 minutae without having to become unicode encoding experts?
The bits and pieces that are specific to unicode are smaller than that, much of the complication of the generator is due to the work required to reduce the size of the trie. The generator is included because we felt that offering a large binary blob for checkin would also run into resistance.
Olaf
Signature · 9 lines
Olaf Weber SGI Phone: +31(0)30-6696796
Veldzigt 2b Fax: +31(0)30-6696799
Technical Lead 3454 PW de Meern Vnet: 955-6796
Storage Software The Netherlands Email: [email protected]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 1 line · Dave Chinner
On Thu, Sep 11, 2014 at 03:37:35PM -0500, Ben Myers wrote:
...
Quoted 20 lines
Quoted 7 lines
When comparing unicode strings for equality, normalization comes into play:
we must compare the normalized forms of strings, not just the raw sequences of bytes. There are a number of defined normalization forms for unicode. We decided on a variant of NFKD we call NFKDI. NFD was chosed over NFC, because calculating NFC requires calculating NFD first, followed by an additional step. NFKD was chosen over NFD because this makes filenames that ought to be equal compare as equal.But are they really equal?
Choosing *compatibility* decomposition over *canonical* decomposition means that compound characters and formatting distinctions don't affect the hash. i.e. "of'fi'ce", "o'ffi'ce" and "office" all hash and compare as the same name, but then they get stored on disk unnormalised. So they are the "same" in memory, but very different on disk.
I note that the unicode spec says this for normalised forms (11.1):
Quoted 7 lines
"A normalized string is guaranteed to be stable; that is, once normalized, a string is normalized according to all future versions of Unicode."
So if we store normalised strings on disk, they are guaranteed to be compatible with all future versions of unicode and anything that goes to use them. So why wouldn't we store normalised forms on disk?
I've had a very similar discussion about normalization in ZFS. Sadly, I can't find where it happened so I can't point you to it. One interesting point that I remember is that storing the original form may be less surprising to an application. Specifically, the name it reads back is the same it supplied during the creation. (Granted, if the file already exists, the application will read back the new form.)
Just FWIW.
Jeff.
Signature · 8 lines
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. - Albert Einstein _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Dave Chinner
Quoted 7 lines
Implementation notes.
Strings are normalized using a trie that stores the relevant information. The trie itself is part of the XFS module, and about 250kB in size. The trie is not checked in: instead we add the source files from the Unicode Character Database and a program that creates the header containing the trie.
Quoted 6 lines
This is rather unappealing. Distros would have to take this code size penalty if they decide one user needs that support. The other millions of users pay that cost even if they don't want it. And then there's validation - how are we supposed to validate that a 250k binary blob is correct and free of issues on every compiler and architecture that the kernel is built on?
The way this needs to be done is to have a separate module for the tables, which XFS or other users then can symbol_get if and only if a mount requires it. The unicode tables should defintively be outside of fs/xfs.
And please run this past lkml or -fsdevel, as people who actually understand unicode and related issues are much more likely to be found there than on the XFS list.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · Olaf Weber
I looked up those discussions in the archives. For example, here's Christoph about rejecting filenames if they're not well-formed unicode.
http://marc.info/?l=linux-fsdevel&m=120876935526856&w=2
And Jamie Lokier making a similar point:http://oss.sgi.com/archives/xfs/2008-04/msg01263.html
And I might now disagree with my past self. While non-ut8 characters are perfectly valid unix filenames, and I think everyones life is easier if we generally stay out of the utf8 business it seems that for this particular use case (shared filesystem with Windows, right) just accepting utf8 should be fine. ZFS is doing, MacOS X apparently is, and NFSv4 requires it, although as far as I know most implementations ignore that requirement.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Olaf Weber
I looked up those discussions in the archives. For example, here's Christoph about rejecting filenames if they're not well-formed unicode.
http://marc.info/?l=linux-fsdevel&m=120876935526856&w=2 And Jamie Lokier making a similar point: http://oss.sgi.com/archives/xfs/2008-04/msg01263.html
Quoted 8 lines
And I might now disagree with my past self. While non-ut8 characters are perfectly valid unix filenames, and I think everyones life is easier if we generally stay out of the utf8 business it seems that for this particular use case (shared filesystem with Windows, right) just accepting utf8 should be fine. ZFS is doing, MacOS X apparently is, and NFSv4 requires it, although as far as I know most implementations ignore that requirement.
One issue is working in environments that are not UTF-8 clean. For example, unpacking a tarball with non-UTF-8 filenames in it. The names would have to be transcoded, which is only really possible if you know the original character set. And if the filesystem flat out rejects non-UTF-8 filenames, then you'd be unable to unpack the tarball at all.
Signature · 9 lines
Olaf Weber SGI Phone: +31(0)30-6696796
Veldzigt 2b Fax: +31(0)30-6696799
Technical Lead 3454 PW de Meern Vnet: 955-6796
Storage Software The Netherlands Email: [email protected]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 17 lines · Olaf Weber
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Olaf Weber
I looked up those discussions in the archives. For example, here's Christoph about rejecting filenames if they're not well-formed unicode.
http://marc.info/?l=linux-fsdevel&m=120876935526856&w=2
And Jamie Lokier making a similar point:http://oss.sgi.com/archives/xfs/2008-04/msg01263.htmlQuoted 8 lines
And I might now disagree with my past self. While non-ut8 characters are perfectly valid unix filenames, and I think everyones life is easier if we generally stay out of the utf8 business it seems that for this particular use case (shared filesystem with Windows, right) just accepting utf8 should be fine. ZFS is doing, MacOS X apparently is, and NFSv4 requires it, although as far as I know most implementations ignore that requirement.
Quoted 6 lines
One issue is working in environments that are not UTF-8 clean. For example, unpacking a tarball with non-UTF-8 filenames in it. The names would have to be transcoded, which is only really possible if you know the original character set. And if the filesystem flat out rejects non-UTF-8 filenames, then you'd be unable to unpack the tarball at all.
So how do existing utf8/unicode enabled filesystems handle this?
I think we should be consistent with ZFS, MacOS and others that already deal with this problem if at all possible. However, this really is a wider policy decision for the kernel/VFS as we want consistent behaviour across all linux filesystems, hence this patchset really needs to discussed at the lkml/-fsdevel level...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Dave Chinner
So how do existing utf8/unicode enabled filesystems handle this?
I think we should be consistent with ZFS, MacOS and others that already deal with this problem if at all possible. However, this really is a wider policy decision for the kernel/VFS as we want consistent behaviour across all linux filesystems, hence this patchset really needs to discussed at the lkml/-fsdevel level...
Absolutely. I've also talked to a few Samba folks at SDC, and one thing they would love to see is conditional case insensitive lookups, e.g.:
- we hash case insensitive with collisions, but perform normal case sensitive lookups.
- with a new AT_CASE_INSENSTIVE flag to the various *at calls that gets passed down to the dcache we enable CI lookups.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hey Gents,
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Dave Chinner
So how do existing utf8/unicode enabled filesystems handle this?
I think we should be consistent with ZFS, MacOS and others that already deal with this problem if at all possible.
Here's a data point from man(zfs):
The following three properties cannot be changed after the file system
is created, and therefore, should be set when the file system is cre-
ated. If the properties are not set with the "zfs create" or "zpool
create" commands, these properties are inherited from the parent
dataset. If the parent dataset lacks these properties due to having
been created prior to these features being supported, the new file sys-
tem will have the default values for these properties.
casesensitivity = sensitive | insensitive | mixed
Indicates whether the file name matching algorithm used by the file
system should be case-sensitive, case-insensitive, or allow a com-
bination of both styles of matching. The default value for the
"casesensitivity" property is "sensitive." Traditionally, UNIX and POSIX file systems have case-sensitive file names.
The "mixed" value for the "casesensitivity" property indicates that
the file system can support requests for both case-sensitive and
case-insensitive matching behavior. Currently, case-insensitive
matching behavior on a file system that supports mixed behavior is
limited to the Solaris CIFS server product. For more information
about the "mixed" value behavior, see the ZFS Administration Guide.
normalization =none | formD | formKCf
Indicates whether the file system should perform a unicode normal-
ization of file names whenever two file names are compared, and
which normalization algorithm should be used. File names are always
stored unmodified, names are normalized as part of any comparison
process. If this property is set to a legal value other than
"none," and the "utf8only" property was left unspecified, the
"utf8only" property is automatically set to "on." The default value
of the "normalization" property is "none." This property cannot be changed after the file system is created.
utf8only =on | off
Indicates whether the file system should reject file names that
include characters that are not present in the UTF-8 character code
set. If this property is explicitly set to "off," the normalization
property must either not be explicitly set or be set to "none." The
default value for the "utf8only" property is "off." This property
cannot be changed after the file system is created.
The "casesensitivity," "normalization," and "utf8only" properties are
also new permissions that can be assigned to non-privileged users by
using the ZFS delegated administration feature.
The original link:
https://www.freebsd.org/cgi/man.cgi?query=zfs&apropos=0&sektion=0&manpath=FreeBSD+8.1-RELEASE&format=html
Quoted 5 lines
Quoted 4 lines
However, this
really is a wider policy decision for the kernel/VFS as we want consistent behaviour across all linux filesystems, hence this patchset really needs to discussed at the lkml/-fsdevel level...
Quoted 8 lines
Absolutely. I've also talked to a few Samba folks at SDC, and one thing they would love to see is conditional case insensitive lookups, e.g.:
- we hash case insensitive with collisions, but perform normal case sensitive lookups.
- with a newAT_CASE_INSENSTIVEflag to the various *at calls that gets passed down to the dcache we enable CI lookups.
I'm working on addressing some of the initial feedback and will be in a position to post for a wider audience later in the week.
Thanks,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs