Skip to contentSkip to conversation

Unicode/UTF-8 support for XFS

RFC In review Ben Myers 3 participants 53 messages Oct 3, 2014 – Oct 14, 2014

Subsystems (4)
lore
Share & tools
lore

Ben Myers

author
#0 lore raw

Hi,

I'm posting this RFC for Unicode support in XFS on Olaf's behalf, as he is busy with other projects. This is the third revision of the series. The others are available here:

http://oss.sgi.com/archives/xfs/2014-09/msg00260.html
http://oss.sgi.com/archives/xfs/2014-09/msg00169.html

In response to feedback for v2, the changes in v3 include:

** trie size is reduced from 245kB to 90kB by using algorithmic
decomposition
* robustness fixes to the trie generator mkutf8data.c
* mkutf8data.c moved to scripts/,
utf8 normalization module moved to lib/, and
compiled under CONFIG_UTF8_NORMALIZATION
* removed CONFIG_XFS_UTF8_DEMAND_LOAD, now it is unconditional
* the unicode version is stored in the superblock, checked at mount time, and passed through to the normalization module.
* add a versioned fsgeometry ioctl, and xfs_info bits to print the unicode version
* TODO don't overload asciici feature bit
* TODO fix patch klunkiness in xfs_da_mount

In response to the initial feedback, the changes in version 2 include:

* linux-fsdevel in the To: line,
* Updated design notes,
* Separation of the fs-independent trie and support code into utf8norm.ko,
* A mechanism for loading the normalization module only when necessary.

I'll post the whole series for completeness sake. Many on -fsdevel will not be interested in the xfs-specific bits, but it may be helpful to have the full series as an example and for testing purposes.

First there is a set of kernel bits, then some libxfs/xfsprogs stuff, and finally a test. (Note: I am not posting the unicode database files due to their large size. There are scripts to download them from unicode.org in the relevant commit headers.)

Thanks,
Ben

Here are Olaf's design notes:

-----------------------------------------------------------------------------
Unicode/UTF-8 support for XFS

So we had a customer request proper unicode support...

* What does "supporting unicode" actually mean?

Quoted 1 line

From a text processing point of view, what a filesystem does with

filenames is simple: it stores and retrieves them, and compares them for equality. It may reject certain byte sequences as invalid filenames (for example, no filename can contain an ASCII NUL).

I've been taking it as a given that when a file is created with a certain byte sequence as its name, then a subsequent directory listing will contain that same byte sequence among the names listed. That things ought to work like that is either obvious to the point of being axiomatic, or -- I suppose -- not. The reader gets one (1) guess as to which camp I'm in.

This leaves comparing names for equality, and in my view this is what "supporting unicode" revolves about.

The present state of affairs is that different byte sequences are different filenames. This amounts to tolerating unicode without actually supporting it.

To support unicode we have to interpret filenames. What happens when (part of) a filename cannot be interpreted? We can reject the filename, interpret the parts we can, or punt and accept it as an uninterpreted blob.

Rejecting ill-formed filenames was my first choice, but I came around on the issue: there are too many ways in which you can end up with having to deal with ill-formed filenames that would leave a user with no recourse but to move whatever they're doing to a different filesystem. Unpacking a tarball with filenames in a different encoding is an example.

Partial interpretation of an ill-formed filename just strikes me as the kind of bad idea that most half-houses are. I admit that I have no stronger objection to this than the fact that it makes the code even more complicated and fragile.

Which leaves "blob" as the preferred option by default for coping with ill-formed filenames.

When comparing well-formed filenames, the question now becomes which byte sequences are considered to be alternative spellings of the same filename. This is where normalization forms come into play, and the unicode standard has quite a bit to say about the subject.

If all you're doing is comparison, then choosing NFD over NFC is easy, because the former is easier to calculate than the latter.

If you want various spellings of "office" to compare equal, then picking NFKD over NFD for comparison is also an obvious choice. (Hand-picking individual compatibility forms is truly a bad idea.) Ways to spell "office": "o_f_f_i_c_e", "o_f_fi_c_e", and "o_ffi_c_e", using no ligatures, the fi ligature, or the ffi ligature. (Some fool thought it a good idea to add these ligatures to unicode, all we get to decide is how to cope.)

The most contentious part is (should be) ignoring the codepoints with the Default_Ignorable_Code_Point property. I've included the list below. My argument, such as it is, is that these code points either have no visible rendering, or in cases like the soft hyphen, are only conditionally visible. The problem with these (as I see it) is that on seeing a filename that might contain them you cannot tell whether they are present. So I propose to ignore them for the purpose of comparing filenames for equality.

Finally, case folding. First of all, it is optional. Then the issue is that you either go the language-specific route, or simplify the task by "just" doing a full casefold (C+F, in unicode parlance). Looking around the net I tend to find that if you're going to do casefolding at all, then a language-independent full casefold is preferred because it is the most predictable option. See
http://www.w3.org/TR/charmod-norm/ for an example of that kind of reasoning.

All of these choices can be argued with, but I do believe that the particular combination of choices I made is a defensible one.

The code refers to these normalization forms as nfkdi and nfkdicf.

* XFS-specific design notes.

XFS uses byte strings for filenames, so UTF-8 is the expected format for unicode filenames. This does raise the question what criteria a byte string must meet to be UTF-8. We settled on the following:
- Valid unicode code points are 0..0x10FFFF, except that
- The surrogates 0xD800..0xDFFF are not valid code points, and
- Valid UTF-8 must be a shortest encoding of a valid unicode code point.

In addition, U+0 (ASCII NUL, '\0') is used to terminate byte strings (and is itself not part of the string). Moreover strings may be length-limited in addition to being NUL-terminated (there is no such thing as an embedded NUL in a length-limited string).

The code uses ("leverages", in corp-speak) the existing XFS infrastructure for case-insensitive filenames. Like the CI code, the name used to create a file is stored on disk, and returned in a lookup. When comparing filenames the normalized forms of the names being compared are generated on the fly from the non-normalized forms stored on disk.

If the borgbit (the bit enabling legacy ASCII-based CI in XFS) is set in the superblock, then case folding is added into the mix. This is the nfkdicf normalization form mentioned above. It allows for the creation of case-insensitive filesystems with UTF-8 support.

* Implementation notes.

Strings are normalized using a trie that stores the relevant information. The trie itself is about 250kB in size, and lives in a separate module. The trie is not checked in: instead we add the source files from the Unicode Character Database and a program that creates the header containing the trie.

The key for a lookup in the trie is a UTF-8 sequence. Each valid UTF-8 sequence leads to a leaf. No invalid sequence does. This means that trie lookups can be used to validate UTF-8 sequences, which why there is no specialized code for the same purpose.

The trie contains information for the version of unicode in which each code point was defined. This matters because non-normalized strings are stored on disk, and newer versions of unicode may introduce new normalized forms. Ideally, the version of unicode used by the filesystem is stored in the filesystem.

The trie also accounts for corrections made in the past to normalizations. This has little value today, because any newly created filesystem would be using unicode version 7.0.0. It is included in order to show, not tell, that such corrections can be handled if they are added in future revisions.

The algorithm used to calculate the sequences of bytes for the normalized form of a UTF-8 string is tricky. The core is found in utf8byte(), with an explanation in the preceeding comment.

The non-XFS-specific supporting code functions have the prefix 'utf8n' if they handle length-limited strings, and 'utf8' if they handle NUL-terminated strings.

----
# Derived Property: Default_Ignorable_Code_Point
# Generated from
# Other_Default_Ignorable_Code_Point

#  + Cf (Format characters)
#  + Variation_Selector
#  - White_Space
#  - FFF9..FFFB (Annotation Characters)
#  - 0600..0605, 06DD, 070F, 110BD (exceptional Cf characters that should be visible)
00AD          ; Default_Ignorable_Code_Point # Cf       SOFT HYPHEN
034F          ; Default_Ignorable_Code_Point # Mn       COMBINING GRAPHEME JOINER
061C          ; Default_Ignorable_Code_Point # Cf       ARABIC LETTER MARK
115F..1160    ; Default_Ignorable_Code_Point # Lo   [2] HANGUL CHOSEONG FILLER..HANGUL JUNGSEONG FILLER
17B4..17B5    ; Default_Ignorable_Code_Point # Mn   [2] KHMER VOWEL INHERENT AQ..KHMER VOWEL INHERENT AA
180B..180D    ; Default_Ignorable_Code_Point # Mn   [3] MONGOLIAN FREE VARIATION SELECTOR ONE..MONGOLIAN FREE VARIATION SELECTOR THREE
180E          ; Default_Ignorable_Code_Point # Cf       MONGOLIAN VOWEL SEPARATOR
200B..200F    ; Default_Ignorable_Code_Point # Cf   [5] ZERO WIDTH SPACE..RIGHT-TO-LEFT MARK
202A..202E    ; Default_Ignorable_Code_Point # Cf   [5] LEFT-TO-RIGHT EMBEDDING..RIGHT-TO-LEFT OVERRIDE
2060..2064    ; Default_Ignorable_Code_Point # Cf   [5] WORD JOINER..INVISIBLE PLUS
2065          ; Default_Ignorable_Code_Point # Cn       <reserved-2065>
2066..206F    ; Default_Ignorable_Code_Point # Cf  [10] LEFT-TO-RIGHT ISOLATE..NOMINAL DIGIT SHAPES
3164          ; Default_Ignorable_Code_Point # Lo       HANGUL FILLER
FE00..FE0F    ; Default_Ignorable_Code_Point # Mn  [16] VARIATION SELECTOR-1..VARIATION SELECTOR-16
FEFF          ; Default_Ignorable_Code_Point # Cf       ZERO WIDTH NO-BREAK SPACE
FFA0          ; Default_Ignorable_Code_Point # Lo       HALFWIDTH HANGUL FILLER
FFF0..FFF8    ; Default_Ignorable_Code_Point # Cn   [9] <reserved-FFF0>..<reserved-FFF8>
1BCA0..1BCA3  ; Default_Ignorable_Code_Point # Cf   [4] SHORTHAND FORMAT LETTER OVERLAP..SHORTHAND FORMAT UP STEP
1D173..1D17A  ; Default_Ignorable_Code_Point # Cf   [8] MUSICAL SYMBOL BEGIN BEAM..MUSICAL SYMBOL END PHRASE
E0000         ; Default_Ignorable_Code_Point # Cn       <reserved-E0000>
E0001         ; Default_Ignorable_Code_Point # Cf       LANGUAGE TAG
E0002..E001F  ; Default_Ignorable_Code_Point # Cn  [30] <reserved-E0002>..<reserved-E001F>
E0020..E007F  ; Default_Ignorable_Code_Point # Cf  [96] TAG SPACE..CANCEL TAG
E0080..E00FF  ; Default_Ignorable_Code_Point # Cn [128] <reserved-E0080>..<reserved-E00FF>
E0100..E01EF  ; Default_Ignorable_Code_Point # Mn [240] VARIATION SELECTOR-17..VARIATION SELECTOR-256
E01F0..E0FFF  ; Default_Ignorable_Code_Point # Cn [3600] <reserved-E01F0>..<reserved-E0FFF>

# Total code points: 4173
----

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#1 lore raw

PATCH 1/16 lib: add unicode character database files

From: Olaf Weber <[email protected]>

Add files from the Unicode Character Database, version 7.0.0, to the source. A helper program that generates a trie used for normalization from these files is part of a separate commit.

---
[v2: Removed large unicode files prior to posting. Get them as below. -bpm] [v3: Moved files to ucd8norm directory. -bpm]
[v4: Moved to lib/ucd. -bpm]

cd lib/ucd
wget http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt
wget http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt
wget http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt
wget http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt
wget http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt
wget http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt
wget http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt
for e in *.txt
do

	base=`basename $e .txt`
	mv $e $base-7.0.0.txt
done
---
 lib/ucd/README | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 lib/ucd/README
lib/ucd/README new file +33 -0
new file mode 100644
index 0000000..d713e66
@@ -0,0 +1,33 @@
+The files in this directory are part of the Unicode Character Database
+for version 7.0.0 of the Unicode standard.
+
+The full set of files can be found here:
+
+  http://www.unicode.org/Public/7.0.0/ucd/
+
+The latest released version of the UCD can be found here:
+
+  http://www.unicode.org/Public/UCD/latest/
+
+The files in this directory are identical, except that they have been
+renamed with a suffix indicating the unicode version.
+
+Individual source links:
+
+  http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt
+  http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt
+  http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt
+  http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt
+  http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt
+  http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt
+  http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt
+
+md5sums
+
+  9a92b2bfe56c6719def926bab524fefd  CaseFolding-7.0.0.txt
+  07b8b1027eb824cf0835314e94f23d2e  DerivedAge-7.0.0.txt
+  90c3340b16821e2f2153acdbe6fc6180  DerivedCombiningClass-7.0.0.txt
+  c41c0601f808116f623de47110ed4f93  DerivedCoreProperties-7.0.0.txt
+  522720ddfc150d8e63a2518634829bce  NormalizationCorrections-7.0.0.txt
+  1f35175eba4a2ad795db489f789ae352  NormalizationTest-7.0.0.txt
+  c8355655731d75e6a3de8c20d7e601ba  UnicodeData-7.0.0.txt
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#2 lore raw

PATCH 2/16 scripts: add trie generator for UTF-8.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

mkutf8data.c is the source for a program that generates utf8data.h, which contains the trie that utf8norm.c uses. The trie is generated from the Unicode 7.0.0 data files. The format of the utf8data[] table is described in utf8norm.c, which is added in the next patch.

---
[v2: the trie is now separated into utf8norm.ko;

     utf8version is now a function and exported;
     introduced CONFIG_XFS_UTF8;
     removed supporting code due to vger size constraint.  --bpm]
[v3: moved trie generator to scripts/;
     introduced CONFIG_UTF8_NORMALIZATION.  --bpm]
---
 lib/Kconfig          |    8 +
 lib/Makefile         |   13 +
 scripts/Makefile     |    1 +
 scripts/mkutf8data.c | 3239 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 3261 insertions(+)
 create mode 100644 scripts/mkutf8data.c
lib/Kconfig +8 -0
index a5ce0c7..c92dfd8 100644
@@ -515,4 +515,12 @@ source "lib/fonts/Kconfig"
 config ARCH_HAS_SG_CHAIN
 	def_bool n
 
+#
+# utf8 normalization module
+#
+config UTF8_NORMALIZATION
+	tristate "UTF-8 normalization support"
+	help
+	  Say Y here to enable utf8 normalization support.
+
 endmenu
lib/Makefile +13 -0
index d6b4bc4..b0b0d57 100644
@@ -196,3 +196,16 @@ quiet_cmd_build_OID_registry = GEN     $@
 clean-files	+= oid_registry_data.c
 
 obj-$(CONFIG_UCS2_STRING) += ucs2_string.o
+
+$(obj)/utf8data.h: $(src)/ucd/*.txt $(objtree)/scripts/mkutf8data FORCE
+	$(call cmd,mkutf8data)
+quiet_cmd_mkutf8data = MKUTF8DATA $@
+      cmd_mkutf8data = $(objtree)/scripts/mkutf8data \
+		-a $(src)/ucd/DerivedAge-7.0.0.txt \
+		-c $(src)/ucd/DerivedCombiningClass-7.0.0.txt \
+		-p $(src)/ucd/DerivedCoreProperties-7.0.0.txt \
+		-d $(src)/ucd/UnicodeData-7.0.0.txt \
+		-f $(src)/ucd/CaseFolding-7.0.0.txt \
+		-n $(src)/ucd/NormalizationCorrections-7.0.0.txt \
+		-t $(src)/ucd/NormalizationTest-7.0.0.txt \
+		-o $@
scripts/Makefile +1 -0
index 72902b5..80fcf43 100644
@@ -16,6 +16,7 @@ hostprogs-$(CONFIG_VT)           += conmakehash
 hostprogs-$(BUILD_C_RECORDMCOUNT) += recordmcount
 hostprogs-$(CONFIG_BUILDTIME_EXTABLE_SORT) += sortextable
 hostprogs-$(CONFIG_ASN1)	 += asn1_compiler
+hostprogs-$(CONFIG_UTF8_NORMALIZATION)	+= mkutf8data
 
 HOSTCFLAGS_sortextable.o = -I$(srctree)/tools/include
 HOSTCFLAGS_asn1_compiler.o = -I$(srctree)/include
scripts/mkutf8data.c new file +3239 -0

2,965 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Part of the diff is not shown here. View the raw message.

author
#3 lore raw

PATCH 3/16 lib: add supporting code for UTF-8.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Supporting functions for UTF-8 normalization are in utf8norm.c with the header utf8norm.h. Two normalization forms are supported: nfkdi and nfkdicf.

nfkdi:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.

nfkdicf:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
- Apply a full casefold (C + F).

For the purposes of the code, a string is valid UTF-8 if:

- The values encoded are 0x1..0x10FFFF.
- The surrogate codepoints 0xD800..0xDFFFF are not encoded.
- The shortest possible encoding is used for all values.

The supporting functions work on null-terminated strings (utf8 prefix) and on length-limited strings (utf8n prefix).

---
[v2: the trie is now separated into utf8norm.ko;

     utf8version is now a function and exported;
     introduced CONFIG_XFS_UTF8;
     removed trie generator due to vger size constraint.  --bpm]
[v3: replaced utf8version with utf8version_is_supported;
     moved utf8norm.ko to lib/ --bpm]
---
 include/linux/utf8norm.h | 116 +++++++++
 lib/Makefile             |   3 +
 lib/utf8norm.c           | 657 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 776 insertions(+)
 create mode 100644 include/linux/utf8norm.h
 create mode 100644 lib/utf8norm.c
include/linux/utf8norm.h new file +116 -0
new file mode 100644
index 0000000..82f86c4
@@ -0,0 +1,116 @@
+/*
+ * Copyright (c) 2014 SGI.
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#ifndef UTF8NORM_H
+#define UTF8NORM_H
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/string.h>
+#include <linux/module.h>
+
+/* An opaque type used to determine the normalization in use. */
+typedef const struct utf8data *utf8data_t;
+
+/* Encoding a unicode version number as a single unsigned int. */
+#define UNICODE_MAJ_SHIFT		(16)
+#define UNICODE_MIN_SHIFT		(8)
+
+#define UNICODE_AGE(MAJ,MIN,REV)			\
+	(((unsigned int)(MAJ) << UNICODE_MAJ_SHIFT) |	\
+	 ((unsigned int)(MIN) << UNICODE_MIN_SHIFT) |	\
+	 ((unsigned int)(REV)))
+
+/* Highest unicode version supported by the data tables. */
+extern int utf8version_is_supported(unsigned int);
+
+/*
+ * Look for the correct utf8data_t for a unicode version.
+ * Returns NULL if the version requested is too new.
+ *
+ * Two normalization forms are supported: nfkdi and nfkdicf.
+ *
+ * nfkdi:
+ *  - Apply unicode normalization form NFKD.
+ *  - Remove any Default_Ignorable_Code_Point.
+ *
+ * nfkdicf:
+ *  - Apply unicode normalization form NFKD.
+ *  - Remove any Default_Ignorable_Code_Point.
+ *  - Apply a full casefold (C + F).
+ */
+extern utf8data_t utf8nfkdi(unsigned int);
+extern utf8data_t utf8nfkdicf(unsigned int);
+
+/*
+ * Determine the maximum age of any unicode character in the string.
+ * Returns 0 if only unassigned code points are present.
+ * Returns -1 if the input is not valid UTF-8.
+ */
+extern int utf8agemax(utf8data_t, const char *);
+extern int utf8nagemax(utf8data_t, const char *, size_t);
+
+/*
+ * Determine the minimum age of any unicode character in the string.
+ * Returns 0 if any unassigned code points are present.
+ * Returns -1 if the input is not valid UTF-8.
+ */
+extern int utf8agemin(utf8data_t, const char *);
+extern int utf8nagemin(utf8data_t, const char *, size_t);
+
+/*
+ * Determine the length of the normalized from of the string,
+ * excluding any terminating NULL byte.
+ * Returns 0 if only ignorable code points are present.
+ * Returns -1 if the input is not valid UTF-8.
+ */
+extern ssize_t utf8len(utf8data_t, const char *);
+extern ssize_t utf8nlen(utf8data_t, const char *, size_t);
+
+/*
+ * Cursor structure used by the normalizer.
+ */
+struct utf8cursor {
+	utf8data_t	data;
+	const char	*s;
+	const char	*p;
+	const char	*ss;
+	const char	*sp;
+	unsigned int	len;
+	unsigned int	slen;
+	short int	ccc;
+	short int	nccc;
+};
+
+/*
+ * Initialize a utf8cursor to normalize a string.
+ * Returns 0 on success.
+ * Returns -1 on failure.
+ */
+extern int utf8cursor(struct utf8cursor *, utf8data_t, const char *);
+extern int utf8ncursor(struct utf8cursor *, utf8data_t, const char *, size_t);
+
+/*
+ * Get the next byte in the normalization.
+ * Returns a value > 0 && < 256 on success.
+ * Returns 0 when the end of the normalization is reached.
+ * Returns -1 if the string being normalized is not valid UTF-8.
+ */
+extern int utf8byte(struct utf8cursor *);
+
+#endif /* UTF8NORM_H */
lib/Makefile +3 -0
index b0b0d57..9e15e2b 100644
@@ -197,6 +197,9 @@ clean-files	+= oid_registry_data.c
 
 obj-$(CONFIG_UCS2_STRING) += ucs2_string.o
 
+obj-$(CONFIG_UTF8_NORMALIZATION) += utf8norm.o
+
+$(obj)/utf8norm.o: $(obj)/utf8data.h
 $(obj)/utf8data.h: $(src)/ucd/*.txt $(objtree)/scripts/mkutf8data FORCE
 	$(call cmd,mkutf8data)
 quiet_cmd_mkutf8data = MKUTF8DATA $@
lib/utf8norm.c new file +657 -0

657 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#4 lore raw

PATCH 4/16 lib/utf8norm.c: reduce the size of utf8data[]

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Remove the Hangul decompositions from the utf8data trie, and do algorithmic decomposition to calculate them on the fly. To store the decomposition the caller of utf8lookup()/utf8nlookup() must provide a 12-byte buffer, which is used to synthesize a leaf with the decomposition. Trie size is reduced from 245kB to 90kB.

This change also contains a number of robustness fixes to the trie generator mkutf8data.c.

---
 include/linux/utf8norm.h |   4 +
 lib/utf8norm.c           | 190 ++++++++++++++++++---
 scripts/mkutf8data.c     | 421 +++++++++++++++++++++++++++++++++++------------
 3 files changed, 492 insertions(+), 123 deletions(-)
include/linux/utf8norm.h +4 -0
index 82f86c4..a6d8ce4 100644
@@ -27,6 +27,9 @@
 /* An opaque type used to determine the normalization in use. */
 typedef const struct utf8data *utf8data_t;
 
+/* Needed in struct utf8cursor below. */
+#define UTF8HANGULLEAF	(12)
+
 /* Encoding a unicode version number as a single unsigned int. */
 #define UNICODE_MAJ_SHIFT		(16)
 #define UNICODE_MIN_SHIFT		(8)
@@ -95,6 +98,7 @@ struct utf8cursor {
 	unsigned int	slen;
 	short int	ccc;
 	short int	nccc;
+	unsigned char	hangul[UTF8HANGULLEAF];
 };
 
 /*
lib/utf8norm.c +169 -21

315 lines not on this page · raw message

scripts/mkutf8data.c +319 -102

800 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#6 lore raw

PATCH 5/16 xfs: return the first match during case-insensitive lookup.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.

---
 fs/xfs/libxfs/xfs_dir2_block.c | 17 +++------
 fs/xfs/libxfs/xfs_dir2_leaf.c  | 37 ++++----------------
 fs/xfs/libxfs/xfs_dir2_node.c  | 79 ++++++++++++++++--------------------------
 fs/xfs/libxfs/xfs_dir2_sf.c    |  8 ++---
 4 files changed, 45 insertions(+), 96 deletions(-)
fs/xfs/libxfs/xfs_dir2_block.c +5 -12
index 9628cec..990bf0c 100644
@@ -725,28 +725,21 @@ xfs_dir2_block_lookup_int(
 		dep = (xfs_dir2_data_entry_t *)
 			((char *)hdr + xfs_dir2_dataptr_to_off(args->geo, addr));
 		/*
-		 * Compare name and if it's an exact match, return the index
-		 * and buffer. If it's the first case-insensitive match, store
-		 * the index and buffer and continue looking for an exact match.
+		 * Compare name and if it's a match, return the
+		 * index and buffer.
 		 */
 		cmp = mp->m_dirnameops->compname(args, dep->name, dep->namelen);
-		if (cmp != XFS_CMP_DIFFERENT && cmp != args->cmpresult) {
+		if (cmp != XFS_CMP_DIFFERENT) {
 			args->cmpresult = cmp;
 			*bpp = bp;
 			*entno = mid;
-			if (cmp == XFS_CMP_EXACT)
-				return 0;
+			return 0;
 		}
 	} while (++mid < be32_to_cpu(btp->count) &&
 			be32_to_cpu(blp[mid].hashval) == hash);
 
 	ASSERT(args->op_flags & XFS_DA_OP_OKNOENT);
-	/*
-	 * Here, we can only be doing a lookup (not a rename or replace).
-	 * If a case-insensitive match was found earlier, return success.
-	 */
-	if (args->cmpresult == XFS_CMP_CASE)
-		return 0;
+	ASSERT(args->cmpresult == XFS_CMP_DIFFERENT);
 	/*
 	 * No match, release the buffer and return ENOENT.
 	 */
fs/xfs/libxfs/xfs_dir2_leaf.c +6 -31
index a19174e..3d572ee 100644
@@ -1226,7 +1226,6 @@ xfs_dir2_leaf_lookup_int(
 	xfs_mount_t		*mp;		/* filesystem mount point */
 	xfs_dir2_db_t		newdb;		/* new data block number */
 	xfs_trans_t		*tp;		/* transaction pointer */
-	xfs_dir2_db_t		cidb = -1;	/* case match data block no. */
 	enum xfs_dacmp		cmp;		/* name compare result */
 	struct xfs_dir2_leaf_entry *ents;
 	struct xfs_dir3_icleaf_hdr leafhdr;
@@ -1290,46 +1289,22 @@ xfs_dir2_leaf_lookup_int(
 						be32_to_cpu(lep->address)));
 		/*
 		 * Compare name and if it's an exact match, return the index
-		 * and buffer. If it's the first case-insensitive match, store
-		 * the index and buffer and continue looking for an exact match.
+		 * and buffer
 		 */
 		cmp = mp->m_dirnameops->compname(args, dep->name, dep->namelen);
-		if (cmp != XFS_CMP_DIFFERENT && cmp != args->cmpresult) {
+		if (cmp != XFS_CMP_DIFFERENT) {
 			args->cmpresult = cmp;
 			*indexp = index;
-			/* case exact match: return the current buffer. */
-			if (cmp == XFS_CMP_EXACT) {
-				*dbpp = dbp;
-				return 0;
-			}
-			cidb = curdb;
+			*dbpp = dbp;
+			return 0;
 		}
 	}
 	ASSERT(args->op_flags & XFS_DA_OP_OKNOENT);
-	/*
-	 * Here, we can only be doing a lookup (not a rename or remove).
-	 * If a case-insensitive match was found earlier, re-read the
-	 * appropriate data block if required and return it.
-	 */
-	if (args->cmpresult == XFS_CMP_CASE) {
-		ASSERT(cidb != -1);
-		if (cidb != curdb) {
-			xfs_trans_brelse(tp, dbp);
-			error = xfs_dir3_data_read(tp, dp,
-					   xfs_dir2_db_to_da(args->geo, cidb),
-					   -1, &dbp);
-			if (error) {
-				xfs_trans_brelse(tp, lbp);
-				return error;
-			}
-		}
-		*dbpp = dbp;
-		return 0;
-	}
+	ASSERT(args->cmpresult == XFS_CMP_DIFFERENT);
+
 	/*
 	 * No match found, return -ENOENT.
 	 */
-	ASSERT(cidb == -1);
 	if (dbp)
 		xfs_trans_brelse(tp, dbp);
 	xfs_trans_brelse(tp, lbp);
fs/xfs/libxfs/xfs_dir2_node.c +30 -49

123 lines not on this page · raw message

fs/xfs/libxfs/xfs_dir2_sf.c +4 -4
index 5079e05..e69fdb7 100644
@@ -757,19 +757,19 @@ xfs_dir2_sf_lookup(
 	for (i = 0, sfep = xfs_dir2_sf_firstentry(sfp); i < sfp->count;
 	     i++, sfep = dp->d_ops->sf_nextentry(sfp, sfep)) {
 		/*
-		 * Compare name and if it's an exact match, return the inode
-		 * number. If it's the first case-insensitive match, store the
-		 * inode number and continue looking for an exact match.
+		 * Compare name and if it's a match, return the inode
+		 * number.
 		 */
 		cmp = dp->i_mount->m_dirnameops->compname(args, sfep->name,
 								sfep->namelen);
-		if (cmp != XFS_CMP_DIFFERENT && cmp != args->cmpresult) {
+		if (cmp != XFS_CMP_DIFFERENT) {
 			args->cmpresult = cmp;
 			args->inumber = dp->d_ops->sf_get_ino(sfp, sfep);
 			args->filetype = dp->d_ops->sf_get_ftype(sfep);
 			if (cmp == XFS_CMP_EXACT)
 				return -EEXIST;
 			ci_sfep = sfep;
+			break;
 		}
 	}
 	ASSERT(args->op_flags & XFS_DA_OP_OKNOENT);
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#11 lore raw

PATCH 6/16 xfs: rename XFS_CMP_CASE to XFS_CMP_MATCH

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Rename XFS_CMP_CASE to XFS_CMP_MATCH. With unicode filenames and normalization, different strings will match on other criteria than case insensitivity.

---
 fs/xfs/libxfs/xfs_da_btree.h  | 2 +-
 fs/xfs/libxfs/xfs_dir2.c      | 9 ++++++---
 fs/xfs/libxfs/xfs_dir2_node.c | 2 +-
 3 files changed, 8 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_da_btree.h +1 -1
index 6e153e3..9ebcc23 100644
@@ -52,7 +52,7 @@ struct xfs_da_geometry {
 enum xfs_dacmp {
 	XFS_CMP_DIFFERENT,	/* names are completely different */
 	XFS_CMP_EXACT,		/* names are exactly the same */
-	XFS_CMP_CASE		/* names are same but differ in case */
+	XFS_CMP_MATCH		/* names are same but differ in encoding */
 };
 
 /*
fs/xfs/libxfs/xfs_dir2.c +6 -3
index 6cef221..32e769b 100644
@@ -74,7 +74,7 @@ xfs_ascii_ci_compname(
 			continue;
 		if (tolower(args->name[i]) != tolower(name[i]))
 			return XFS_CMP_DIFFERENT;
-		result = XFS_CMP_CASE;
+		result = XFS_CMP_MATCH;
 	}
 
 	return result;
@@ -315,8 +315,11 @@ xfs_dir_cilookup_result(
 {
 	if (args->cmpresult == XFS_CMP_DIFFERENT)
 		return -ENOENT;
-	if (args->cmpresult != XFS_CMP_CASE ||
-					!(args->op_flags & XFS_DA_OP_CILOOKUP))
+	if (args->cmpresult == XFS_CMP_EXACT)
+		return -EEXIST;
+	ASSERT(args->cmpresult == XFS_CMP_MATCH);
+	/* Only dup the found name if XFS_DA_OP_CILOOKUP is set. */
+	if (!(args->op_flags & XFS_DA_OP_CILOOKUP))
 		return -EEXIST;
 
 	args->value = kmem_alloc(len, KM_NOFS | KM_MAYFAIL);
fs/xfs/libxfs/xfs_dir2_node.c +1 -1
index 1778c40..9d46e8d 100644
@@ -2023,7 +2023,7 @@ xfs_dir2_node_lookup(
 	error = xfs_da3_node_lookup_int(state, &rval);
 	if (error)
 		rval = error;
-	else if (rval == -ENOENT && args->cmpresult == XFS_CMP_CASE) {
+	else if (rval == -ENOENT && args->cmpresult == XFS_CMP_MATCH) {
 		/* If a CI match, dup the actual name and return -EEXIST */
 		xfs_dir2_data_entry_t	*dep;
 
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#12 lore raw

PATCH 7/16 xfs: add xfs_nameops.normhash

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Add a normhash callout to the xfs_nameops. This callout takes an xfs_da_args structure as its argument, and calculates a hash value over the name. It may in the process create a normalized form of the name, and assign that to the norm/normlen fields in the xfs_da_args structure.

---
 fs/xfs/libxfs/xfs_da_btree.c |  9 +++++++++
 fs/xfs/libxfs/xfs_da_btree.h |  3 +++
 fs/xfs/libxfs/xfs_dir2.c     | 42 +++++++++++++++++++++++++++++++++++++-----
 3 files changed, 49 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_da_btree.c +9 -0
index 2c42ae2..07a3acf 100644
@@ -1990,8 +1990,17 @@ xfs_default_hashname(
 	return xfs_da_hashname(name->name, name->len);
 }
 
+STATIC int
+xfs_da_normhash(
+	struct xfs_da_args *args)
+{
+	args->hashval = xfs_da_hashname(args->name, args->namelen);
+	return 0;
+}
+
 const struct xfs_nameops xfs_default_nameops = {
 	.hashname	= xfs_default_hashname,
+	.normhash	= xfs_da_normhash,
 	.compname	= xfs_da_compname
 };
 
fs/xfs/libxfs/xfs_da_btree.h +3 -0
index 9ebcc23..6cdafee 100644
@@ -61,7 +61,9 @@ enum xfs_dacmp {
 typedef struct xfs_da_args {
 	struct xfs_da_geometry *geo;	/* da block geometry */
 	const __uint8_t	*name;		/* string (maybe not NULL terminated) */
+	const __uint8_t	*norm;		/* normalized name (may be NULL) */
 	int		namelen;	/* length of string (maybe no NULL) */
+	int		normlen;	/* length of normalized name */
 	__uint8_t	filetype;	/* filetype of inode for directories */
 	__uint8_t	*value;		/* set of bytes (maybe contain NULLs) */
 	int		valuelen;	/* length of value */
@@ -150,6 +152,7 @@ typedef struct xfs_da_state {
  */
 struct xfs_nameops {
 	xfs_dahash_t	(*hashname)(struct xfs_name *);
+	int		(*normhash)(struct xfs_da_args *);
 	enum xfs_dacmp	(*compname)(struct xfs_da_args *,
 					const unsigned char *, int);
 };
fs/xfs/libxfs/xfs_dir2.c +37 -5

143 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#13 lore raw

PATCH 8/16 xfs: change interface of xfs_nameops.hashname

In reply to Ben Myers

From: Olaf Weber <[email protected]>

With the introduction of the xfs_nameops.normhash callout, all uses of the hashname callout now occur in places where an xfs_name structure must be explicitly created just to match the parameter passing convention of this callout. Change the arguments to a const unsigned char * and int instead.

[v2: pass a 3rd argument for sb_utf8version to hashname. --bpm]

---
 fs/xfs/libxfs/xfs_da_btree.c   | 18 ++++++++++--------
 fs/xfs/libxfs/xfs_da_btree.h   |  3 ++-
 fs/xfs/libxfs/xfs_dir2.c       |  8 +++++---
 fs/xfs/libxfs/xfs_dir2_block.c |  4 +++-
 fs/xfs/libxfs/xfs_dir2_data.c  |  4 +++-
 5 files changed, 23 insertions(+), 14 deletions(-)
fs/xfs/libxfs/xfs_da_btree.c +10 -8
index 07a3acf..ec6cc98 100644
@@ -1973,6 +1973,15 @@ xfs_da_hashname(const __uint8_t *name, int namelen)
 	}
 }
 
+xfs_dahash_t
+xfs_da_hashname_op(
+	const __uint8_t		*name,
+	int 			namelen,
+	unsigned int 		unused)
+{
+	return xfs_da_hashname(name, namelen);
+}
+
 enum xfs_dacmp
 xfs_da_compname(
 	struct xfs_da_args *args,
@@ -1983,13 +1992,6 @@ xfs_da_compname(
 					XFS_CMP_EXACT : XFS_CMP_DIFFERENT;
 }
 
-static xfs_dahash_t
-xfs_default_hashname(
-	struct xfs_name	*name)
-{
-	return xfs_da_hashname(name->name, name->len);
-}
-
 STATIC int
 xfs_da_normhash(
 	struct xfs_da_args *args)
@@ -1999,7 +2001,7 @@ xfs_da_normhash(
 }
 
 const struct xfs_nameops xfs_default_nameops = {
-	.hashname	= xfs_default_hashname,
+	.hashname	= xfs_da_hashname_op,
 	.normhash	= xfs_da_normhash,
 	.compname	= xfs_da_compname
 };
fs/xfs/libxfs/xfs_da_btree.h +2 -1
index 6cdafee..ce6888a 100644
@@ -151,7 +151,8 @@ typedef struct xfs_da_state {
  * Name ops for directory and/or attr name operations
  */
 struct xfs_nameops {
-	xfs_dahash_t	(*hashname)(struct xfs_name *);
+	xfs_dahash_t	(*hashname)(const unsigned char *, int,
+					unsigned int);
 	int		(*normhash)(struct xfs_da_args *);
 	enum xfs_dacmp	(*compname)(struct xfs_da_args *,
 					const unsigned char *, int);
fs/xfs/libxfs/xfs_dir2.c +5 -3
index 55733a6..4eb0973 100644
@@ -45,13 +45,15 @@ struct xfs_name xfs_name_dotdot = { (unsigned char *)"..", 2, XFS_DIR3_FT_DIR };
  */
 STATIC xfs_dahash_t
 xfs_ascii_ci_hashname(
-	struct xfs_name	*name)
+	const unsigned char *name,
+	int len,
+	unsigned int unused)
 {
 	xfs_dahash_t	hash;
 	int		i;
 
-	for (i = 0, hash = 0; i < name->len; i++)
-		hash = tolower(name->name[i]) ^ rol32(hash, 7);
+	for (i = 0, hash = 0; i < len; i++)
+		hash = tolower(name[i]) ^ rol32(hash, 7);
 
 	return hash;
 }
fs/xfs/libxfs/xfs_dir2_block.c +3 -1
index 990bf0c..12ebdd8 100644
@@ -1231,7 +1231,9 @@ xfs_dir2_sf_to_block(
 		name.name = sfep->name;
 		name.len = sfep->namelen;
 		blp[2 + i].hashval = cpu_to_be32(mp->m_dirnameops->
-							hashname(&name));
+					hashname(sfep->name,
+						 sfep->namelen,
+						 0 /* version for later */));
 		blp[2 + i].address = cpu_to_be32(xfs_dir2_byte_to_dataptr(
 						 (char *)dep - (char *)hdr));
 		offset = (int)((char *)(tagp + 1) - (char *)hdr);
fs/xfs/libxfs/xfs_dir2_data.c +3 -1
index fdd803f..25b0f7b 100644
@@ -179,7 +179,9 @@ __xfs_dir3_data_check(
 						((char *)dep - (char *)hdr));
 			name.name = dep->name;
 			name.len = dep->namelen;
-			hash = mp->m_dirnameops->hashname(&name);
+			hash = mp->m_dirnameops->hashname(dep->name,
+					dep->namelen,
+					0 /* version for later */);
 			for (i = 0; i < be32_to_cpu(btp->count); i++) {
 				if (be32_to_cpu(lep[i].address) == addr &&
 				    be32_to_cpu(lep[i].hashval) == hash)
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#16 lore raw

PATCH 9/16 xfs: add a superblock feature bit to indicate UTF-8 support.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

When UTF-8 support is enabled, the xfs_dir_ci_inode_operations must be installed. Add xfs_sb_version_hasci(), which tests both the borgbit and the utf8bit, and returns true if at least one of them is set. Replace calls to xfs_sb_version_hasasciici() as needed.

---
 fs/xfs/libxfs/xfs_sb.h | 24 +++++++++++++++++++++++-
 fs/xfs/xfs_fs.h        |  1 +
 fs/xfs/xfs_fsops.c     |  4 +++-
 fs/xfs/xfs_iops.c      |  4 ++--
 4 files changed, 29 insertions(+), 4 deletions(-)
fs/xfs/libxfs/xfs_sb.h +23 -1
index 2e73970..525eacb 100644
@@ -70,6 +70,7 @@ struct xfs_trans;
 #define XFS_SB_VERSION2_RESERVED4BIT	0x00000004
 #define XFS_SB_VERSION2_ATTR2BIT	0x00000008	/* Inline attr rework */
 #define XFS_SB_VERSION2_PARENTBIT	0x00000010	/* parent pointers */
+#define XFS_SB_VERSION2_UTF8BIT		0x00000020      /* utf8 names */
 #define XFS_SB_VERSION2_PROJID32BIT	0x00000080	/* 32 bit project id */
 #define XFS_SB_VERSION2_CRCBIT		0x00000100	/* metadata CRCs */
 #define XFS_SB_VERSION2_FTYPE		0x00000200	/* inode type in dir */
@@ -77,6 +78,7 @@ struct xfs_trans;
 #define	XFS_SB_VERSION2_OKBITS		\
 	(XFS_SB_VERSION2_LAZYSBCOUNTBIT	| \
 	 XFS_SB_VERSION2_ATTR2BIT	| \
+	 XFS_SB_VERSION2_UTF8BIT	| \
 	 XFS_SB_VERSION2_PROJID32BIT	| \
 	 XFS_SB_VERSION2_FTYPE)
 
@@ -509,8 +511,10 @@ xfs_sb_has_ro_compat_feature(
 }
 
 #define XFS_SB_FEAT_INCOMPAT_FTYPE	(1 << 0)	/* filetype in dirent */
+#define XFS_SB_FEAT_INCOMPAT_UTF8	(1 << 1)	/* utf-8 name support */
 #define XFS_SB_FEAT_INCOMPAT_ALL \
-		(XFS_SB_FEAT_INCOMPAT_FTYPE)
+		(XFS_SB_FEAT_INCOMPAT_FTYPE | \
+		 XFS_SB_FEAT_INCOMPAT_UTF8)
 
 #define XFS_SB_FEAT_INCOMPAT_UNKNOWN	~XFS_SB_FEAT_INCOMPAT_ALL
 static inline bool
@@ -558,6 +562,24 @@ static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp)
 		(sbp->sb_features_ro_compat & XFS_SB_FEAT_RO_COMPAT_FINOBT);
 }
 
+static inline int xfs_sb_version_hasutf8(xfs_sb_t *sbp)
+{
+	return (XFS_SB_VERSION_NUM(sbp) == XFS_SB_VERSION_5 &&
+		xfs_sb_has_incompat_feature(sbp, XFS_SB_FEAT_INCOMPAT_UTF8)) ||
+		(xfs_sb_version_hasmorebits(sbp) &&
+		(sbp->sb_features2 & XFS_SB_VERSION2_UTF8BIT));
+}
+
+/*
+ * Special case: there are a number of places where we need to test
+ * both the borgbit and the utf8bit, and take the same action if
+ * either of those is set.
+ */
+static inline int xfs_sb_version_hasci(xfs_sb_t *sbp)
+{
+	return xfs_sb_version_hasasciici(sbp) || xfs_sb_version_hasutf8(sbp);
+}
+
 /*
  * end of superblock version macros
  */
fs/xfs/xfs_fs.h +1 -0
index 18dc721..e845d75 100644
@@ -239,6 +239,7 @@ typedef struct xfs_fsop_resblks {
 #define XFS_FSOP_GEOM_FLAGS_V5SB	0x8000	/* version 5 superblock */
 #define XFS_FSOP_GEOM_FLAGS_FTYPE	0x10000	/* inode directory types */
 #define XFS_FSOP_GEOM_FLAGS_FINOBT	0x20000	/* free inode btree */
+#define XFS_FSOP_GEOM_FLAGS_UTF8	0x40000	/* utf8 filenames */
 
 /*
  * Minimum and maximum sizes need for growth checks.
fs/xfs/xfs_fsops.c +3 -1
index f91de1e..1a83eef 100644
@@ -103,7 +103,9 @@ xfs_fs_geometry(
 			(xfs_sb_version_hasftype(&mp->m_sb) ?
 				XFS_FSOP_GEOM_FLAGS_FTYPE : 0) |
 			(xfs_sb_version_hasfinobt(&mp->m_sb) ?
-				XFS_FSOP_GEOM_FLAGS_FINOBT : 0);
+				XFS_FSOP_GEOM_FLAGS_FINOBT : 0) |
+			(xfs_sb_version_hasutf8(&mp->m_sb) ?
+				XFS_FSOP_GEOM_FLAGS_UTF8 : 0);
 		geo->logsectsize = xfs_sb_version_hassector(&mp->m_sb) ?
 				mp->m_sb.sb_logsectsize : BBSIZE;
 		geo->rtsectsize = mp->m_sb.sb_blocksize;
fs/xfs/xfs_iops.c +2 -2
index 7212949..cea3d64 100644
@@ -335,9 +335,9 @@ xfs_vn_unlink(
 	/*
 	 * With unlink, the VFS makes the dentry "negative": no inode,
 	 * but still hashed. This is incompatible with case-insensitive
-	 * mode, so invalidate (unhash) the dentry in CI-mode.
+	 * or utf8 mode, so invalidate (unhash) the dentry in CI-mode.
 	 */
-	if (xfs_sb_version_hasasciici(&XFS_M(dir->i_sb)->m_sb))
+	if (xfs_sb_version_hasci(&XFS_M(dir->i_sb)->m_sb))
 		d_invalidate(dentry);
 	return 0;
 }
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#19 lore raw

PATCH 10/16 xfs: store utf8version in the superblock

In reply to Ben Myers

From: Ben Myers <[email protected]>

The utf8 version a filesystem was created with needs to be stored in order that normalizations will remain stable over the lifetime of the filesystem. Convert sb_pad to sb_utf8version in the super block. This also adds checks at mount time to see whether the unicode normalization module has support for the version of unicode that the filesystem requires. If not we fail the mount.

---
 fs/xfs/libxfs/xfs_dir2.c | 28 ++++++++++++++++---
 fs/xfs/libxfs/xfs_sb.c   |  4 +--
 fs/xfs/libxfs/xfs_sb.h   | 10 ++++---
 fs/xfs/libxfs/xfs_utf8.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_utf8.h | 24 +++++++++++++++++
 5 files changed, 126 insertions(+), 10 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_utf8.c
 create mode 100644 fs/xfs/libxfs/xfs_utf8.h
fs/xfs/libxfs/xfs_dir2.c +24 -4
index 4eb0973..2c89211 100644
@@ -157,10 +157,30 @@ xfs_da_mount(
 				(uint)sizeof(xfs_da_node_entry_t);
 	dageo->magicpct = (dageo->blksize * 37) / 100;
 
-	if (xfs_sb_version_hasasciici(&mp->m_sb))
-		mp->m_dirnameops = &xfs_ascii_ci_nameops;
-	else
-		mp->m_dirnameops = &xfs_default_nameops;
+	if (xfs_sb_version_hasutf8(&mp->m_sb)) {
+#ifdef CONFIG_XFS_UTF8
+		if (!xfs_utf8_version_ok(mp))
+			return -ENOSYS;
+
+		/* XXX these are replaced in the next patch need
+		   to do some kind of reordering here */
+		if (xfs_sb_version_hasasciici(&mp->m_sb))
+			mp->m_dirnameops = &xfs_ascii_ci_nameops;
+		else
+			mp->m_dirnameops = &xfs_default_nameops;
+#else
+		xfs_warn(mp,
+"Recompile XFS with CONFIG_XFS_UTF8 to mount this filesystem");
+		kmem_free(mp->m_dir_geo);
+		kmem_free(mp->m_attr_geo);
+		return -ENOSYS;
+#endif
+	} else {
+		if (xfs_sb_version_hasasciici(&mp->m_sb))
+			mp->m_dirnameops = &xfs_ascii_ci_nameops;
+		else
+			mp->m_dirnameops = &xfs_default_nameops;
+	}
 
 	return 0;
 }
fs/xfs/libxfs/xfs_sb.c +2 -2
index ad525a5..1ee7d33 100644
@@ -99,7 +99,7 @@ static const struct {
 	{ offsetof(xfs_sb_t, sb_features_incompat),	0 },
 	{ offsetof(xfs_sb_t, sb_features_log_incompat),	0 },
 	{ offsetof(xfs_sb_t, sb_crc),		0 },
-	{ offsetof(xfs_sb_t, sb_pad),		0 },
+	{ offsetof(xfs_sb_t, sb_utf8version),	0 },
 	{ offsetof(xfs_sb_t, sb_pquotino),	0 },
 	{ offsetof(xfs_sb_t, sb_lsn),		0 },
 	{ sizeof(xfs_sb_t),			0 }
@@ -443,7 +443,7 @@ __xfs_sb_from_disk(
 	to->sb_features_incompat = be32_to_cpu(from->sb_features_incompat);
 	to->sb_features_log_incompat =
 				be32_to_cpu(from->sb_features_log_incompat);
-	to->sb_pad = 0;
+	to->sb_utf8version = be32_to_cpu(from->sb_utf8version);
 	to->sb_pquotino = be64_to_cpu(from->sb_pquotino);
 	to->sb_lsn = be64_to_cpu(from->sb_lsn);
 	/* Convert on-disk flags to in-memory flags? */
fs/xfs/libxfs/xfs_sb.h +6 -4
index 525eacb..dc7b6c6 100644
@@ -159,7 +159,7 @@ typedef struct xfs_sb {
 	__uint32_t	sb_features_log_incompat;
 
 	__uint32_t	sb_crc;		/* superblock crc */
-	__uint32_t	sb_pad;
+	__uint32_t	sb_utf8version;	/* unicode version */
 
 	xfs_ino_t	sb_pquotino;	/* project quota inode */
 	xfs_lsn_t	sb_lsn;		/* last write sequence */
@@ -245,7 +245,7 @@ typedef struct xfs_dsb {
 	__be32		sb_features_log_incompat;
 
 	__le32		sb_crc;		/* superblock crc */
-	__be32		sb_pad;
+	__be32		sb_utf8version;	/* version of unicode */
 
 	__be64		sb_pquotino;	/* project quota inode */
 	__be64		sb_lsn;		/* last write sequence */
@@ -271,7 +271,7 @@ typedef enum {
 	XFS_SBS_LOGSECTLOG, XFS_SBS_LOGSECTSIZE, XFS_SBS_LOGSUNIT,
 	XFS_SBS_FEATURES2, XFS_SBS_BAD_FEATURES2, XFS_SBS_FEATURES_COMPAT,
 	XFS_SBS_FEATURES_RO_COMPAT, XFS_SBS_FEATURES_INCOMPAT,
-	XFS_SBS_FEATURES_LOG_INCOMPAT, XFS_SBS_CRC, XFS_SBS_PAD,
+	XFS_SBS_FEATURES_LOG_INCOMPAT, XFS_SBS_CRC, XFS_SBS_UTF8VERSION,
 	XFS_SBS_PQUOTINO, XFS_SBS_LSN,
 	XFS_SBS_FIELDCOUNT
 } xfs_sb_field_t;
@@ -303,6 +303,7 @@ typedef enum {
 #define XFS_SB_FEATURES_INCOMPAT XFS_SB_MVAL(FEATURES_INCOMPAT)
 #define XFS_SB_FEATURES_LOG_INCOMPAT XFS_SB_MVAL(FEATURES_LOG_INCOMPAT)
 #define XFS_SB_CRC		XFS_SB_MVAL(CRC)
+#define XFS_SB_UTF8VERSION	XFS_SB_MVAL(UTF8VERSION)
 #define XFS_SB_PQUOTINO		XFS_SB_MVAL(PQUOTINO)
 #define	XFS_SB_NUM_BITS		((int)XFS_SBS_FIELDCOUNT)
 #define	XFS_SB_ALL_BITS		((1LL << XFS_SB_NUM_BITS) - 1)
@@ -313,7 +314,8 @@ typedef enum {
 	 XFS_SB_ICOUNT | XFS_SB_IFREE | XFS_SB_FDBLOCKS | XFS_SB_FEATURES2 | \
 	 XFS_SB_BAD_FEATURES2 | XFS_SB_FEATURES_COMPAT | \
 	 XFS_SB_FEATURES_RO_COMPAT | XFS_SB_FEATURES_INCOMPAT | \
-	 XFS_SB_FEATURES_LOG_INCOMPAT | XFS_SB_PQUOTINO)
+	 XFS_SB_FEATURES_LOG_INCOMPAT | XFS_SB_UTF8VERSION | \
+	 XFS_SB_PQUOTINO)
 
 
 /*
fs/xfs/libxfs/xfs_utf8.c new file +70 -0

70 lines not on this page · raw message

fs/xfs/libxfs/xfs_utf8.h new file +24 -0
new file mode 100644
index 0000000..8a700de
@@ -0,0 +1,24 @@
+/*
+ * Copyright (c) 2014 SGI.
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#ifndef XFS_UTF8_H
+#define XFS_UTF8_H
+
+extern int xfs_utf8_version_ok(struct xfs_mount *);
+
+#endif /* XFS_UTF8_H */
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#21 lore raw

PATCH 11/16 xfs: add xfs_nameops for utf8 and utf8+casefold.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

The xfs_utf8_nameops use the nfkdi normalization when comparing filenames, and are installed if the utf8bit is set in the super block.

The xfs_utf8_ci_nameops use the nfkdicf normalization when comparing filenames, and are installed if both the utf8bit and the borgbit are set in the superblock.

Normalized filenames are not stored on disk. Normalization will fail if a filename is not valid UTF-8, in which case the filename is treated as an opaque blob.

---

[v2: updated to use utf8norm.ko module;
     compiled conditionally on CONFIG_XFS_UTF8=y;
     utf8version is now a function;
     move xfs_utf8.[ch] into libxfs. --bpm]
[v3: pass utf8version from the superblock through xfs_nameops
     instead of the max version of the normalization module. --bpm]
---
 fs/xfs/Kconfig           |   9 ++
 fs/xfs/Makefile          |   2 +
 fs/xfs/libxfs/xfs_dir2.c |   4 +-
 fs/xfs/libxfs/xfs_utf8.c | 208 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_utf8.h |   3 +
 fs/xfs/xfs_iops.c        |   2 +-
 6 files changed, 225 insertions(+), 3 deletions(-)
fs/xfs/Kconfig +9 -0
index 5d47b4d..1e8a463 100644
@@ -95,3 +95,12 @@ config XFS_DEBUG
 	  not useful unless you are debugging a particular problem.
 
 	  Say N unless you are an XFS developer, or you play one on TV.
+
+config XFS_UTF8
+	bool "XFS UTF-8 support"
+	depends on XFS_FS
+	select CONFIG_UTF8_NORMALIZATION
+	help
+	  Say Y here to enable utf8 normalization support in XFS.  You
+	  will be able to mount and use filesystems created with the
+	  utf8 mkfs.xfs option.
fs/xfs/Makefile +2 -0
index d617999..192aaca 100644
@@ -114,6 +114,8 @@ xfs-$(CONFIG_XFS_QUOTA)		+= xfs_dquot.o \
 				   xfs_qm.o \
 				   xfs_quotaops.o
 
+xfs-$(CONFIG_XFS_UTF8)		+= libxfs/xfs_utf8.o
+
 # xfs_rtbitmap is shared with libxfs
 xfs-$(CONFIG_XFS_RT)		+= xfs_rtalloc.o
 
fs/xfs/libxfs/xfs_dir2.c +2 -2
index 2c89211..9cfbd6b 100644
@@ -165,9 +165,9 @@ xfs_da_mount(
 		/* XXX these are replaced in the next patch need
 		   to do some kind of reordering here */
 		if (xfs_sb_version_hasasciici(&mp->m_sb))
-			mp->m_dirnameops = &xfs_ascii_ci_nameops;
+			mp->m_dirnameops = &xfs_utf8_ci_nameops;
 		else
-			mp->m_dirnameops = &xfs_default_nameops;
+			mp->m_dirnameops = &xfs_utf8_nameops;
 #else
 		xfs_warn(mp,
 "Recompile XFS with CONFIG_XFS_UTF8 to mount this filesystem");
fs/xfs/libxfs/xfs_utf8.c +208 -0

211 lines not on this page · raw message

fs/xfs/libxfs/xfs_utf8.h +3 -0
index 8a700de..404db54 100644
@@ -21,4 +21,7 @@
 
 extern int xfs_utf8_version_ok(struct xfs_mount *);
 
+extern struct xfs_nameops xfs_utf8_nameops;
+extern struct xfs_nameops xfs_utf8_ci_nameops;
+
 #endif /* XFS_UTF8_H */
fs/xfs/xfs_iops.c +1 -1
index cea3d64..fbfb1bb 100644
@@ -1257,7 +1257,7 @@ xfs_setup_inode(
 		break;
 	case S_IFDIR:
 		lockdep_set_class(&ip->i_lock.mr_lock, &xfs_dir_ilock_class);
-		if (xfs_sb_version_hasasciici(&XFS_M(inode->i_sb)->m_sb))
+		if (xfs_sb_version_hasci(&XFS_M(inode->i_sb)->m_sb))
 			inode->i_op = &xfs_dir_ci_inode_operations;
 		else
 			inode->i_op = &xfs_dir_inode_operations;
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#23 lore raw

PATCH 12/16 xfs: apply utf-8 normalization rules to user extended attribute names

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Apply the same rules for UTF-8 normalization to the names of user-defined extended attributes. System attributes are excluded because they are not user-visible in the first place, and the kernel is expected to know what it is doing when naming them.

---
 fs/xfs/libxfs/xfs_attr.c      | 56 ++++++++++++++++++++++++++++++++++++-------
 fs/xfs/libxfs/xfs_attr_leaf.c | 11 +++++++--
 fs/xfs/libxfs/xfs_utf8.c      |  7 ++++++
 fs/xfs/xfs_attr_list.c        | 12 +++++++++-
 4 files changed, 75 insertions(+), 11 deletions(-)
fs/xfs/libxfs/xfs_attr.c +48 -8
index 353fb42..68e7ce3 100644
@@ -83,12 +83,14 @@ xfs_attr_args_init(
 	const unsigned char	*name,
 	int			flags)
 {
+	struct xfs_mount	*mp = dp->i_mount;
+	int			error;
 
 	if (!name)
 		return -EINVAL;
 
 	memset(args, 0, sizeof(*args));
-	args->geo = dp->i_mount->m_attr_geo;
+	args->geo = mp->m_attr_geo;
 	args->whichfork = XFS_ATTR_FORK;
 	args->dp = dp;
 	args->flags = flags;
@@ -97,7 +99,11 @@ xfs_attr_args_init(
 	if (args->namelen >= MAXNAMELEN)
 		return -EFAULT;		/* match IRIX behaviour */
 
-	args->hashval = xfs_da_hashname(args->name, args->namelen);
+	if (!xfs_sb_version_hasutf8(&mp->m_sb))
+		args->hashval = xfs_da_hashname(args->name, args->namelen);
+	else if ((error = mp->m_dirnameops->normhash(args)) != 0)
+		return error;
+
 	return 0;
 }
 
@@ -154,6 +160,9 @@ xfs_attr_get(
 		error = xfs_attr_node_get(&args);
 	xfs_iunlock(ip, lock_mode);
 
+	if (args.norm)
+		kmem_free(args.norm);
+
 	*valuelenp = args.valuelen;
 	return error == -EEXIST ? 0 : error;
 }
@@ -216,8 +225,11 @@ xfs_attr_set(
 		return -EIO;
 
 	error = xfs_attr_args_init(&args, dp, name, flags);
-	if (error)
+	if (error) {
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
+	}
 
 	args.value = value;
 	args.valuelen = valuelen;
@@ -227,8 +239,11 @@ xfs_attr_set(
 	args.total = xfs_attr_calc_size(&args, &local);
 
 	error = xfs_qm_dqattach(dp, 0);
-	if (error)
+	if (error) {
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
+	}
 
 	/*
 	 * If the inode doesn't have an attribute fork, add one.
@@ -239,8 +254,11 @@ xfs_attr_set(
 			XFS_ATTR_SF_ENTSIZE_BYNAME(args.namelen, valuelen);
 
 		error = xfs_bmap_add_attrfork(dp, sf_size, rsvd);
-		if (error)
+		if (error) {
+			if (args.norm)
+				kmem_free(args.norm);
 			return error;
+		}
 	}
 
 	/*
@@ -270,6 +288,8 @@ xfs_attr_set(
 	error = xfs_trans_reserve(args.trans, &tres, args.total, 0);
 	if (error) {
 		xfs_trans_cancel(args.trans, 0);
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
 	}
 	xfs_ilock(dp, XFS_ILOCK_EXCL);
@@ -280,6 +300,8 @@ xfs_attr_set(
 	if (error) {
 		xfs_iunlock(dp, XFS_ILOCK_EXCL);
 		xfs_trans_cancel(args.trans, XFS_TRANS_RELEASE_LOG_RES);
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
 	}
 
@@ -327,6 +349,8 @@ xfs_attr_set(
 						 XFS_TRANS_RELEASE_LOG_RES);
 			xfs_iunlock(dp, XFS_ILOCK_EXCL);
 
+			if (args.norm)
+				kmem_free(args.norm);
 			return error ? error : err2;
 		}
 
@@ -388,7 +412,8 @@ xfs_attr_set(
 	xfs_trans_log_inode(args.trans, dp, XFS_ILOG_CORE);
 	error = xfs_trans_commit(args.trans, XFS_TRANS_RELEASE_LOG_RES);
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
-
+	if (args.norm)
+		kmem_free(args.norm);
 	return error;
 
 out:
@@ -397,6 +422,8 @@ out:
 			XFS_TRANS_RELEASE_LOG_RES|XFS_TRANS_ABORT);
 	}
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free(args.norm);
 	return error;
 }
 
@@ -425,8 +452,11 @@ xfs_attr_remove(
 		return -ENOATTR;
 
 	error = xfs_attr_args_init(&args, dp, name, flags);
-	if (error)
+	if (error) {
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
+	}
 
 	args.firstblock = &firstblock;
 	args.flist = &flist;
@@ -439,8 +469,11 @@ xfs_attr_remove(
 	args.op_flags = XFS_DA_OP_OKNOENT;
 
 	error = xfs_qm_dqattach(dp, 0);
-	if (error)
+	if (error) {
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
+	}
 
 	/*
 	 * Start our first transaction of the day.
@@ -466,6 +499,8 @@ xfs_attr_remove(
 				  XFS_ATTRRM_SPACE_RES(mp), 0);
 	if (error) {
 		xfs_trans_cancel(args.trans, 0);
+		if (args.norm)
+			kmem_free(args.norm);
 		return error;
 	}
 
@@ -506,6 +541,8 @@ xfs_attr_remove(
 	xfs_trans_log_inode(args.trans, dp, XFS_ILOG_CORE);
 	error = xfs_trans_commit(args.trans, XFS_TRANS_RELEASE_LOG_RES);
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free(args.norm);
 
 	return error;
 
@@ -515,6 +552,9 @@ out:
 			XFS_TRANS_RELEASE_LOG_RES|XFS_TRANS_ABORT);
 	}
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free(args.norm);
+
 	return error;
 }
 
fs/xfs/libxfs/xfs_attr_leaf.c +9 -2
index b1f73db..c991a88 100644
@@ -661,6 +661,7 @@ int
 xfs_attr_shortform_to_leaf(xfs_da_args_t *args)
 {
 	xfs_inode_t *dp;
+	struct xfs_mount *mp;
 	xfs_attr_shortform_t *sf;
 	xfs_attr_sf_entry_t *sfe;
 	xfs_da_args_t nargs;
@@ -673,6 +674,7 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args)
 	trace_xfs_attr_sf_to_leaf(args);
 
 	dp = args->dp;
+	mp = dp->i_mount;
 	ifp = dp->i_afp;
 	sf = (xfs_attr_shortform_t *)ifp->if_u1.if_data;
 	size = be16_to_cpu(sf->hdr.totsize);
@@ -726,13 +728,18 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args)
 		nargs.namelen = sfe->namelen;
 		nargs.value = &sfe->nameval[nargs.namelen];
 		nargs.valuelen = sfe->valuelen;
-		nargs.hashval = xfs_da_hashname(sfe->nameval,
-						sfe->namelen);
 		nargs.flags = XFS_ATTR_NSP_ONDISK_TO_ARGS(sfe->flags);
+		if (!xfs_sb_version_hasutf8(&mp->m_sb))
+			nargs.hashval = xfs_da_hashname(sfe->nameval,
+							sfe->namelen);
+		else if ((error = mp->m_dirnameops->normhash(&nargs)) != 0)
+			goto out;
 		error = xfs_attr3_leaf_lookup_int(bp, &nargs); /* set a->index */
 		ASSERT(error == -ENOATTR);
 		error = xfs_attr3_leaf_add(bp, &nargs);
 		ASSERT(error != -ENOSPC);
+		if (nargs.norm)
+			kmem_free(nargs.norm);
 		if (error)
 			goto out;
 		sfe = XFS_ATTR_SF_NEXTENTRY(sfe);
fs/xfs/libxfs/xfs_utf8.c +7 -0
index 1e75299..ede6228 100644
@@ -38,6 +38,7 @@
 #include "xfs_inode.h"
 #include "xfs_inode_item.h"
 #include "xfs_bmap.h"
+#include "xfs_attr.h"
 #include "xfs_error.h"
 #include "xfs_trace.h"
 #include "xfs_utf8.h"
@@ -109,6 +110,9 @@ xfs_utf8_normhash(
 	unsigned int	sb_utf8version =
 		args->dp->i_mount->m_sb.sb_utf8version;
 
+	/* Don't normalize system attribute names. */
+	if (args->flags & (ATTR_ROOT|ATTR_SECURE))
+		goto blob;
 	nfkdi = utf8nfkdi(sb_utf8version);
 	/* Failure to normalize is treated as a blob. */
 	if ((normlen = utf8nlen(nfkdi, args->name, args->namelen)) < 0)
@@ -213,6 +217,9 @@ xfs_utf8_ci_normhash(
 	unsigned int	sb_utf8version =
 		args->dp->i_mount->m_sb.sb_utf8version;
 
+	/* Don't normalize system attribute names. */
+	if (args->flags & (ATTR_ROOT|ATTR_SECURE))
+		goto blob;
 	nfkdicf = utf8nfkdicf(sb_utf8version);
 	/* Failure to normalize is treated as a blob. */
 	if ((normlen = utf8nlen(nfkdicf, args->name, args->namelen)) < 0)
fs/xfs/xfs_attr_list.c +11 -1
index 62db83a..034199d 100644
@@ -76,12 +76,14 @@ xfs_attr_shortform_list(xfs_attr_list_context_t *context)
 	xfs_attr_shortform_t *sf;
 	xfs_attr_sf_entry_t *sfe;
 	xfs_inode_t *dp;
+	struct xfs_mount *mp;
 	int sbsize, nsbuf, count, i;
 	int error;
 
 	ASSERT(context != NULL);
 	dp = context->dp;
 	ASSERT(dp != NULL);
+	mp = dp->i_mount;
 	ASSERT(dp->i_afp != NULL);
 	sf = (xfs_attr_shortform_t *)dp->i_afp->if_u1.if_data;
 	ASSERT(sf != NULL);
@@ -154,7 +156,15 @@ xfs_attr_shortform_list(xfs_attr_list_context_t *context)
 		}
 
 		sbp->entno = i;
-		sbp->hash = xfs_da_hashname(sfe->nameval, sfe->namelen);
+
+		/* ATTR_ROOT and ATTR_SECURE are never normalized. */
+		if (!xfs_sb_version_hasutf8(&mp->m_sb) ||
+		    (sfe->flags & (ATTR_ROOT|ATTR_SECURE))) {
+			sbp->hash = xfs_da_hashname(sfe->nameval, sfe->namelen);
+		} else {
+			sbp->hash = mp->m_dirnameops->hashname(sfe->nameval,
+				       sfe->namelen, mp->m_sb.sb_utf8version);
+		}
 		sbp->name = sfe->nameval;
 		sbp->namelen = sfe->namelen;
 		/* These are bytes, and both on-disk, don't endian-flip */
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#24 lore raw

PATCH 13/16 xfs: implement demand load of utf8norm.ko

In reply to Ben Myers

From: Ben Myers <[email protected]>

The utf8 normalization module is large and there is no need to have it loaded unless an xfs filesystem with utf8 enabled has been mounted. This loads utf8norm.ko at mount time for filesystems that need it.

---
[v2: updated for utf8version_is_supported. --bpm]
[v3: removed CONFIG_XFS_UTF8_DEMAND_LOAD. --bpm]

---
 fs/xfs/libxfs/xfs_dir2.c |   9 +++++
 fs/xfs/libxfs/xfs_utf8.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_utf8.h |   3 ++
 fs/xfs/xfs_super.c       |   6 +++
 4 files changed, 118 insertions(+)
fs/xfs/libxfs/xfs_dir2.c +9 -0
index 9cfbd6b..844044b 100644
@@ -35,6 +35,9 @@
 #include "xfs_error.h"
 #include "xfs_trace.h"
 #include "xfs_dinode.h"
+#ifdef CONFIG_XFS_UTF8
+#include "xfs_utf8.h"
+#endif
 
 struct xfs_name xfs_name_dotdot = { (unsigned char *)"..", 2, XFS_DIR3_FT_DIR };
 
@@ -159,6 +162,12 @@ xfs_da_mount(
 
 	if (xfs_sb_version_hasutf8(&mp->m_sb)) {
 #ifdef CONFIG_XFS_UTF8
+		if (xfs_init_utf8_module(mp)) {
+			kmem_free(mp->m_dir_geo);
+			kmem_free(mp->m_attr_geo);
+			return -ENOSYS;
+		}
+
 		if (!xfs_utf8_version_ok(mp))
 			return -ENOSYS;
 
fs/xfs/libxfs/xfs_utf8.c +100 -0

106 lines not on this page · raw message

fs/xfs/libxfs/xfs_utf8.h +3 -0
index 404db54..b79ce05 100644
@@ -24,4 +24,7 @@ extern int xfs_utf8_version_ok(struct xfs_mount *);
 extern struct xfs_nameops xfs_utf8_nameops;
 extern struct xfs_nameops xfs_utf8_ci_nameops;
 
+extern int xfs_init_utf8_module(struct xfs_mount *);
+extern void xfs_put_utf8_module(void);
+
 #endif /* XFS_UTF8_H */
fs/xfs/xfs_super.c +6 -0
index b194652..60a3ebc 100644
@@ -47,6 +47,9 @@
 #include "xfs_dinode.h"
 #include "xfs_filestream.h"
 #include "xfs_quota.h"
+#ifdef CONFIG_XFS_UTF8
+#include "xfs_utf8.h"
+#endif
 
 #include <linux/namei.h>
 #include <linux/init.h>
@@ -1809,6 +1812,9 @@ exit_xfs_fs(void)
 	xfs_mru_cache_uninit();
 	xfs_destroy_workqueues();
 	xfs_destroy_zones();
+#ifdef CONFIG_XFS_UTF8
+	xfs_put_utf8_module();
+#endif
 }
 
 module_init(init_xfs_fs);
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#27 lore raw

PATCH 14/16 xfs: rename XFS_IOC_FSGEOM to XFS_IOC_FSGEOM_V2

In reply to Ben Myers

From: Ben Myers <[email protected]>

We'll be creating a new versioned XFS_IOC_FSGEOMETRY ioctl and structure so rename the current revision to _V2.

---
 fs/xfs/xfs_fs.h      |  8 ++++----
 fs/xfs/xfs_fsops.c   |  2 +-
 fs/xfs/xfs_fsops.h   |  3 ++-
 fs/xfs/xfs_ioctl.c   | 12 ++++++------
 fs/xfs/xfs_ioctl32.c |  4 ++--
 5 files changed, 15 insertions(+), 14 deletions(-)
fs/xfs/xfs_fs.h +4 -4
index e845d75..fd45cbe 100644
@@ -180,9 +180,9 @@ typedef struct xfs_fsop_geom_v1 {
 } xfs_fsop_geom_v1_t;
 
 /*
- * Output for XFS_IOC_FSGEOMETRY
+ * Output for XFS_IOC_FSGEOMETRY_V2
  */
-typedef struct xfs_fsop_geom {
+typedef struct xfs_fsop_geom_v2 {
 	__u32		blocksize;	/* filesystem (data) block size */
 	__u32		rtextsize;	/* realtime extent size		*/
 	__u32		agblocks;	/* fsblocks in an AG		*/
@@ -204,7 +204,7 @@ typedef struct xfs_fsop_geom {
 	__u32		rtsectsize;	/* realtime sector size, bytes	*/
 	__u32		dirblocksize;	/* directory block size, bytes	*/
 	__u32		logsunit;	/* log stripe unit, bytes */
-} xfs_fsop_geom_t;
+} xfs_fsop_geom_v2_t;
 
 /* Output for XFS_FS_COUNTS */
 typedef struct xfs_fsop_counts {
@@ -555,7 +555,7 @@ typedef struct xfs_swapext
 #define XFS_IOC_FSSETDM_BY_HANDLE    _IOW ('X', 121, struct xfs_fsop_setdm_handlereq)
 #define XFS_IOC_ATTRLIST_BY_HANDLE   _IOW ('X', 122, struct xfs_fsop_attrlist_handlereq)
 #define XFS_IOC_ATTRMULTI_BY_HANDLE  _IOW ('X', 123, struct xfs_fsop_attrmulti_handlereq)
-#define XFS_IOC_FSGEOMETRY	     _IOR ('X', 124, struct xfs_fsop_geom)
+#define XFS_IOC_FSGEOMETRY_V2	     _IOR ('X', 124, struct xfs_fsop_geom_v2)
 #define XFS_IOC_GOINGDOWN	     _IOR ('X', 125, __uint32_t)
 /*	XFS_IOC_GETFSUUID ---------- deprecated 140	 */
 
fs/xfs/xfs_fsops.c +1 -1
index 1a83eef..b69468e 100644
@@ -50,7 +50,7 @@
 int
 xfs_fs_geometry(
 	xfs_mount_t		*mp,
-	xfs_fsop_geom_t		*geo,
+	xfs_fsop_geom_v2_t	*geo,
 	int			new_version)
 {
 
fs/xfs/xfs_fsops.h +2 -1
index 1b6a98b..26e7343 100644
@@ -18,7 +18,8 @@
 #ifndef __XFS_FSOPS_H__
 #define	__XFS_FSOPS_H__
 
-extern int xfs_fs_geometry(xfs_mount_t *mp, xfs_fsop_geom_t *geo, int nversion);
+extern int xfs_fs_geometry(xfs_mount_t *mp, xfs_fsop_geom_v2_t *geo,
+		int nversion);
 extern int xfs_growfs_data(xfs_mount_t *mp, xfs_growfs_data_t *in);
 extern int xfs_growfs_log(xfs_mount_t *mp, xfs_growfs_log_t *in);
 extern int xfs_fs_counts(xfs_mount_t *mp, xfs_fsop_counts_t *cnt);
fs/xfs/xfs_ioctl.c +6 -6
index 3799695..4393405 100644
@@ -823,7 +823,7 @@ xfs_ioc_fsgeometry_v1(
 	xfs_mount_t		*mp,
 	void			__user *arg)
 {
-	xfs_fsop_geom_t         fsgeo;
+	xfs_fsop_geom_v2_t	fsgeo;
 	int			error;
 
 	error = xfs_fs_geometry(mp, &fsgeo, 3);
@@ -841,18 +841,18 @@ xfs_ioc_fsgeometry_v1(
 }
 
 STATIC int
-xfs_ioc_fsgeometry(
+xfs_ioc_fsgeometry_v2(
 	xfs_mount_t		*mp,
 	void			__user *arg)
 {
-	xfs_fsop_geom_t		fsgeo;
+	xfs_fsop_geom_v2_t	fsgeo;
 	int			error;
 
 	error = xfs_fs_geometry(mp, &fsgeo, 4);
 	if (error)
 		return error;
 
-	if (copy_to_user(arg, &fsgeo, sizeof(fsgeo)))
+	if (copy_to_user(arg, &fsgeo, sizeof(xfs_fsop_geom_v2_t)))
 		return -EFAULT;
 	return 0;
 }
@@ -1564,8 +1564,8 @@ xfs_file_ioctl(
 	case XFS_IOC_FSGEOMETRY_V1:
 		return xfs_ioc_fsgeometry_v1(mp, arg);
 
-	case XFS_IOC_FSGEOMETRY:
-		return xfs_ioc_fsgeometry(mp, arg);
+	case XFS_IOC_FSGEOMETRY_V2:
+		return xfs_ioc_fsgeometry_v2(mp, arg);
 
 	case XFS_IOC_GETVERSION:
 		return put_user(inode->i_generation, (int __user *)arg);
fs/xfs/xfs_ioctl32.c +2 -2
index a554646..207b224 100644
@@ -64,7 +64,7 @@ xfs_compat_ioc_fsgeometry_v1(
 	struct xfs_mount	  *mp,
 	compat_xfs_fsop_geom_v1_t __user *arg32)
 {
-	xfs_fsop_geom_t		  fsgeo;
+	xfs_fsop_geom_v2_t	  fsgeo;
 	int			  error;
 
 	error = xfs_fs_geometry(mp, &fsgeo, 3);
@@ -543,7 +543,7 @@ xfs_file_compat_ioctl(
 	switch (cmd) {
 	/* No size or alignment issues on any arch */
 	case XFS_IOC_DIOINFO:
-	case XFS_IOC_FSGEOMETRY:
+	case XFS_IOC_FSGEOMETRY_V2:
 	case XFS_IOC_FSGETXATTR:
 	case XFS_IOC_FSSETXATTR:
 	case XFS_IOC_FSGETXATTRA:
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#30 lore raw

PATCH 15/16 xfs: xfs_fs_geometry returns a number of bytes to copy

In reply to Ben Myers

From: Ben Myers <[email protected]>

The versioned xfs_fsop_geom_t will be of variable size. Make xfs_fs_geometry return the number of bytes to copy out to userspace for a given version of the structure.

---
 fs/xfs/xfs_fsops.c   | 6 ++----
 fs/xfs/xfs_fsops.h   | 2 +-
 fs/xfs/xfs_ioctl.c   | 6 ++++--
 fs/xfs/xfs_ioctl32.c | 3 ++-
 4 files changed, 9 insertions(+), 8 deletions(-)
fs/xfs/xfs_fsops.c +2 -4
index b69468e..cf87e16 100644
@@ -51,11 +51,9 @@ int
 xfs_fs_geometry(
 	xfs_mount_t		*mp,
 	xfs_fsop_geom_v2_t	*geo,
-	int			new_version)
+	int			new_version,
+	size_t			*bytes)
 {
-
-	memset(geo, 0, sizeof(*geo));
-
 	geo->blocksize = mp->m_sb.sb_blocksize;
 	geo->rtextsize = mp->m_sb.sb_rextsize;
 	geo->agblocks = mp->m_sb.sb_agblocks;
fs/xfs/xfs_fsops.h +1 -1
index 26e7343..74e1fee 100644
@@ -19,7 +19,7 @@
 #define	__XFS_FSOPS_H__
 
 extern int xfs_fs_geometry(xfs_mount_t *mp, xfs_fsop_geom_v2_t *geo,
-		int nversion);
+		int new_version, size_t *bytes);
 extern int xfs_growfs_data(xfs_mount_t *mp, xfs_growfs_data_t *in);
 extern int xfs_growfs_log(xfs_mount_t *mp, xfs_growfs_log_t *in);
 extern int xfs_fs_counts(xfs_mount_t *mp, xfs_fsop_counts_t *cnt);
fs/xfs/xfs_ioctl.c +4 -2
index 4393405..1657ce5 100644
@@ -826,7 +826,8 @@ xfs_ioc_fsgeometry_v1(
 	xfs_fsop_geom_v2_t	fsgeo;
 	int			error;
 
-	error = xfs_fs_geometry(mp, &fsgeo, 3);
+	memset(&fsgeo, 0, sizeof(fsgeo));
+	error = xfs_fs_geometry(mp, &fsgeo, 3, NULL);
 	if (error)
 		return error;
 
@@ -848,7 +849,8 @@ xfs_ioc_fsgeometry_v2(
 	xfs_fsop_geom_v2_t	fsgeo;
 	int			error;
 
-	error = xfs_fs_geometry(mp, &fsgeo, 4);
+	memset(&fsgeo, 0, sizeof(fsgeo));
+	error = xfs_fs_geometry(mp, &fsgeo, 4, NULL);
 	if (error)
 		return error;
 
fs/xfs/xfs_ioctl32.c +2 -1
index 207b224..aca988a 100644
@@ -67,7 +67,8 @@ xfs_compat_ioc_fsgeometry_v1(
 	xfs_fsop_geom_v2_t	  fsgeo;
 	int			  error;
 
-	error = xfs_fs_geometry(mp, &fsgeo, 3);
+	memset(&fsgeo, 0, sizeof(fsgeo));
+	error = xfs_fs_geometry(mp, &fsgeo, 3, NULL);
 	if (error)
 		return error;
 	/* The 32-bit variant simply has some padding at the end */
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#32 lore raw

PATCH 16/16 xfs: add versioned fsgeom ioctl with utf8version field

In reply to Ben Myers

From: Ben Myers <[email protected]>

This adds a utf8version field to the xfs_fs_geom structure. An important characteristic of this version of the ioctl is that fsgeo.version needs to be set by the caller to specify which version of the structure to return.

---
 fs/xfs/xfs_fs.h    | 31 +++++++++++++++++++++++++++++++
 fs/xfs/xfs_fsops.c | 13 ++++++++++++-
 fs/xfs/xfs_fsops.h |  2 +-
 fs/xfs/xfs_ioctl.c | 31 +++++++++++++++++++++++++++++++
 4 files changed, 75 insertions(+), 2 deletions(-)
fs/xfs/xfs_fs.h +31 -0
index fd45cbe..2f4d430 100644
@@ -206,6 +206,34 @@ typedef struct xfs_fsop_geom_v2 {
 	__u32		logsunit;	/* log stripe unit, bytes */
 } xfs_fsop_geom_v2_t;
 
+/*
+ * Output for XFS_IOC_FSGEOMETRY
+ */
+typedef struct xfs_fsop_geom {
+	__u32		blocksize;	/* filesystem (data) block size */
+	__u32		rtextsize;	/* realtime extent size		*/
+	__u32		agblocks;	/* fsblocks in an AG		*/
+	__u32		agcount;	/* number of allocation groups	*/
+	__u32		logblocks;	/* fsblocks in the log		*/
+	__u32		sectsize;	/* (data) sector size, bytes	*/
+	__u32		inodesize;	/* inode size in bytes		*/
+	__u32		imaxpct;	/* max allowed inode space(%)	*/
+	__u64		datablocks;	/* fsblocks in data subvolume	*/
+	__u64		rtblocks;	/* fsblocks in realtime subvol	*/
+	__u64		rtextents;	/* rt extents in realtime subvol*/
+	__u64		logstart;	/* starting fsblock of the log	*/
+	unsigned char	uuid[16];	/* unique id of the filesystem	*/
+	__u32		sunit;		/* stripe unit, fsblocks	*/
+	__u32		swidth;		/* stripe width, fsblocks	*/
+	__s32		version;	/* structure version		*/
+	__u32		flags;		/* superblock version flags	*/
+	__u32		logsectsize;	/* log sector size, bytes	*/
+	__u32		rtsectsize;	/* realtime sector size, bytes	*/
+	__u32		dirblocksize;	/* directory block size, bytes	*/
+	__u32		logsunit;	/* log stripe unit, bytes */
+	__u32		utf8version;	/* Unicode version		*/
+} xfs_fsop_geom_t;
+
 /* Output for XFS_FS_COUNTS */
 typedef struct xfs_fsop_counts {
 	__u64	freedata;	/* free data section blocks */
@@ -221,6 +249,8 @@ typedef struct xfs_fsop_resblks {
 } xfs_fsop_resblks_t;
 
 #define XFS_FSOP_GEOM_VERSION	0
+/* skipped 1-4 to match existing new_version xfs_fs_geometry argument */
+#define XFS_FSOP_GEOM_VERSION5	5
 
 #define XFS_FSOP_GEOM_FLAGS_ATTR	0x0001	/* attributes in use	*/
 #define XFS_FSOP_GEOM_FLAGS_NLINK	0x0002	/* 32-bit nlink values	*/
@@ -557,6 +587,7 @@ typedef struct xfs_swapext
 #define XFS_IOC_ATTRMULTI_BY_HANDLE  _IOW ('X', 123, struct xfs_fsop_attrmulti_handlereq)
 #define XFS_IOC_FSGEOMETRY_V2	     _IOR ('X', 124, struct xfs_fsop_geom_v2)
 #define XFS_IOC_GOINGDOWN	     _IOR ('X', 125, __uint32_t)
+#define XFS_IOC_FSGEOMETRY	     _IOR ('X', 126, struct xfs_fsop_geom)
 /*	XFS_IOC_GETFSUUID ---------- deprecated 140	 */
 
 
fs/xfs/xfs_fsops.c +12 -1
index cf87e16..d70acf8 100644
@@ -50,10 +50,12 @@
 int
 xfs_fs_geometry(
 	xfs_mount_t		*mp,
-	xfs_fsop_geom_v2_t	*geo,
+	void			*buffer,
 	int			new_version,
 	size_t			*bytes)
 {
+	xfs_fsop_geom_t		*geo = (xfs_fsop_geom_t *)buffer;
+
 	geo->blocksize = mp->m_sb.sb_blocksize;
 	geo->rtextsize = mp->m_sb.sb_rextsize;
 	geo->agblocks = mp->m_sb.sb_agblocks;
@@ -115,6 +117,15 @@ xfs_fs_geometry(
 				XFS_FSOP_GEOM_FLAGS_LOGV2 : 0);
 		geo->logsunit = mp->m_sb.sb_logsunit;
 	}
+	if (new_version >= XFS_FSOP_GEOM_VERSION5) {
+		geo->version = XFS_FSOP_GEOM_VERSION5;
+		geo->flags |= (xfs_sb_version_hasutf8(&mp->m_sb) ?
+				XFS_FSOP_GEOM_FLAGS_UTF8 : 0);
+		geo->utf8version = mp->m_sb.sb_utf8version;
+		if (bytes)
+			*bytes = sizeof(xfs_fsop_geom_v2_t) +
+				 sizeof(geo->utf8version);
+	}
 	return 0;
 }
 
fs/xfs/xfs_fsops.h +1 -1
index 74e1fee..b723f36 100644
@@ -18,7 +18,7 @@
 #ifndef __XFS_FSOPS_H__
 #define	__XFS_FSOPS_H__
 
-extern int xfs_fs_geometry(xfs_mount_t *mp, xfs_fsop_geom_v2_t *geo,
+extern int xfs_fs_geometry(xfs_mount_t *mp, void *buffer,
 		int new_version, size_t *bytes);
 extern int xfs_growfs_data(xfs_mount_t *mp, xfs_growfs_data_t *in);
 extern int xfs_growfs_log(xfs_mount_t *mp, xfs_growfs_log_t *in);
fs/xfs/xfs_ioctl.c +31 -0
index 1657ce5..6308680 100644
@@ -859,6 +859,34 @@ xfs_ioc_fsgeometry_v2(
 	return 0;
 }
 
+STATIC int
+xfs_ioc_fsgeometry(
+	struct xfs_mount	*mp,
+	void			__user *arg)
+{
+	xfs_fsop_geom_t		fsgeo;
+	int			version, error;
+	size_t			bytes;
+
+	/* offsetof(version)? XXX just get 32 bits? */
+	if (copy_from_user(&fsgeo, arg, sizeof(xfs_fsop_geom_v1_t)))
+		return -EFAULT;
+
+	version = fsgeo.version;
+
+	if (version < XFS_FSOP_GEOM_VERSION5)
+		return -EINVAL;
+
+	memset(&fsgeo, 0, sizeof(fsgeo));
+	error = xfs_fs_geometry(mp, &fsgeo, version, &bytes);
+	if (error)
+		return error;
+
+	if (copy_to_user(arg, &fsgeo, bytes))
+		return -EFAULT;
+	return 0;
+}
+
 /*
  * Linux extended inode flags interface.
  */
@@ -1569,6 +1597,9 @@ xfs_file_ioctl(
 	case XFS_IOC_FSGEOMETRY_V2:
 		return xfs_ioc_fsgeometry_v2(mp, arg);
 
+	case XFS_IOC_FSGEOMETRY:
+		return xfs_ioc_fsgeometry(mp, arg);
+
 	case XFS_IOC_GETVERSION:
 		return put_user(inode->i_generation, (int __user *)arg);
 
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#34 lore raw

PATCH 17/35 xfsprogs: add unicode character database files

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Add files from the Unicode Character Database, version 7.0.0, to the source. A helper program that generates a trie used for normalization from these files is part of a separate commit.

[v2: moved from support to utf8norm/ucd -bpm]

---
 utf8norm/ucd/README | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 utf8norm/ucd/README
utf8norm/ucd/README new file +33 -0
new file mode 100644
index 0000000..d713e66
@@ -0,0 +1,33 @@
+The files in this directory are part of the Unicode Character Database
+for version 7.0.0 of the Unicode standard.
+
+The full set of files can be found here:
+
+  http://www.unicode.org/Public/7.0.0/ucd/
+
+The latest released version of the UCD can be found here:
+
+  http://www.unicode.org/Public/UCD/latest/
+
+The files in this directory are identical, except that they have been
+renamed with a suffix indicating the unicode version.
+
+Individual source links:
+
+  http://www.unicode.org/Public/7.0.0/ucd/CaseFolding.txt
+  http://www.unicode.org/Public/7.0.0/ucd/DerivedAge.txt
+  http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedCombiningClass.txt
+  http://www.unicode.org/Public/7.0.0/ucd/DerivedCoreProperties.txt
+  http://www.unicode.org/Public/7.0.0/ucd/NormalizationCorrections.txt
+  http://www.unicode.org/Public/7.0.0/ucd/NormalizationTest.txt
+  http://www.unicode.org/Public/7.0.0/ucd/UnicodeData.txt
+
+md5sums
+
+  9a92b2bfe56c6719def926bab524fefd  CaseFolding-7.0.0.txt
+  07b8b1027eb824cf0835314e94f23d2e  DerivedAge-7.0.0.txt
+  90c3340b16821e2f2153acdbe6fc6180  DerivedCombiningClass-7.0.0.txt
+  c41c0601f808116f623de47110ed4f93  DerivedCoreProperties-7.0.0.txt
+  522720ddfc150d8e63a2518634829bce  NormalizationCorrections-7.0.0.txt
+  1f35175eba4a2ad795db489f789ae352  NormalizationTest-7.0.0.txt
+  c8355655731d75e6a3de8c20d7e601ba  UnicodeData-7.0.0.txt
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#35 lore raw

PATCH 18/35 xfsprogs: add trie generator for UTF-8.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

mkutf8data.c is the source for a program that generates utf8data.h, which contains the trie that utf8norm.c uses. The trie is generated from the Unicode 7.0.0 data files. The format of the utf8data[] table is described in utf8norm.c, which is added in the next patch.

[v2: moved to utf8norm directory. -bpm]

---
 Makefile              |    2 +-
 utf8norm/Makefile     |   24 +
 utf8norm/mkutf8data.c | 3239 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 3264 insertions(+), 1 deletion(-)
 create mode 100644 utf8norm/Makefile
 create mode 100644 utf8norm/mkutf8data.c
Makefile +1 -1
index f56aebd..74778b5 100644
@@ -40,7 +40,7 @@ LDIRDIRT = $(SRCDIR)
 LDIRT += $(SRCTAR)
 endif
 
-LIB_SUBDIRS = libxfs libxlog libxcmd libhandle libdisk
+LIB_SUBDIRS = utf8norm libxfs libxlog libxcmd libhandle libdisk
 TOOL_SUBDIRS = copy db estimate fsck fsr growfs io logprint mkfs quota \
 		mdrestore repair rtcp m4 man doc po debian
 
utf8norm/Makefile new file +24 -0
new file mode 100644
index 0000000..a32660e
@@ -0,0 +1,24 @@
+#
+# Copyright (c) 2014 SGI. All Rights Reserved.
+#
+
+TOPDIR = ..
+include $(TOPDIR)/include/builddefs
+
+default = ../include/utf8data.h
+
+../include/utf8data.h: mkutf8data.c
+	cc -o mkutf8data mkutf8data.c
+	cd ucd; ../mkutf8data
+	mv ucd/utf8data.h ../include
+
+default clean:
+	rm -f mkutf8data ../include/utf8data.h
+
+default install:
+
+default install-dev:
+
+default install-qa:
+
+-include .ltdep
utf8norm/mkutf8data.c new file +3239 -0

2,968 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Part of the diff is not shown here. View the raw message.

author
#36 lore raw

PATCH 19/35 xfsprogs: add supporting code for UTF-8.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Supporting functions for UTF-8 normalization are in utf8norm.c with the header utf8norm.h. Two normalization forms are supported: nfkdi and nfkdicf.

nfkdi:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.

nfkdicf:
- Apply unicode normalization form NFKD.
- Remove any Default_Ignorable_Code_Point.
- Apply a full casefold (C + F).

For the purposes of the code, a string is valid UTF-8 if:

- The values encoded are 0x1..0x10FFFF.
- The surrogate codepoints 0xD800..0xDFFFF are not encoded.
- The shortest possible encoding is used for all values.

The supporting functions work on null-terminated strings (utf8 prefix) and on length-limited strings (utf8n prefix).

[v2: synced from kernel version, minor updates to remove/add includes,
and remove linux module exports. --bpm]

---
 include/utf8norm.h | 111 +++++++++
 libxfs/Makefile    |   1 +
 libxfs/utf8norm.c  | 643 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 755 insertions(+)
 create mode 100644 include/utf8norm.h
 create mode 100644 libxfs/utf8norm.c
include/utf8norm.h new file +111 -0
new file mode 100644
index 0000000..cd77580
@@ -0,0 +1,111 @@
+/*
+ * Copyright (c) 2014 SGI.
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#ifndef UTF8NORM_H
+#define UTF8NORM_H
+
+/* An opaque type used to determine the normalization in use. */
+typedef const struct utf8data *utf8data_t;
+
+/* Encoding a unicode version number as a single unsigned int. */
+#define UNICODE_MAJ_SHIFT		(16)
+#define UNICODE_MIN_SHIFT		(8)
+
+#define UNICODE_AGE(MAJ,MIN,REV)			\
+	(((unsigned int)(MAJ) << UNICODE_MAJ_SHIFT) |	\
+	 ((unsigned int)(MIN) << UNICODE_MIN_SHIFT) |	\
+	 ((unsigned int)(REV)))
+
+/* Highest unicode version supported by the data tables. */
+extern int utf8version_is_supported(unsigned int);
+
+/*
+ * Look for the correct utf8data_t for a unicode version.
+ * Returns NULL if the version requested is too new.
+ *
+ * Two normalization forms are supported: nfkdi and nfkdicf.
+ *
+ * nfkdi:
+ *  - Apply unicode normalization form NFKD.
+ *  - Remove any Default_Ignorable_Code_Point.
+ *
+ * nfkdicf:
+ *  - Apply unicode normalization form NFKD.
+ *  - Remove any Default_Ignorable_Code_Point.
+ *  - Apply a full casefold (C + F).
+ */
+extern utf8data_t utf8nfkdi(unsigned int);
+extern utf8data_t utf8nfkdicf(unsigned int);
+
+/*
+ * Determine the maximum age of any unicode character in the string.
+ * Returns 0 if only unassigned code points are present.
+ * Returns -1 if the input is not valid UTF-8.
+ */
+extern int utf8agemax(utf8data_t, const char *);
+extern int utf8nagemax(utf8data_t, const char *, size_t);
+
+/*
+ * Determine the minimum age of any unicode character in the string.
+ * Returns 0 if any unassigned code points are present.
+ * Returns -1 if the input is not valid UTF-8.
+ */
+extern int utf8agemin(utf8data_t, const char *);
+extern int utf8nagemin(utf8data_t, const char *, size_t);
+
+/*
+ * Determine the length of the normalized from of the string,
+ * excluding any terminating NULL byte.
+ * Returns 0 if only ignorable code points are present.
+ * Returns -1 if the input is not valid UTF-8.
+ */
+extern ssize_t utf8len(utf8data_t, const char *);
+extern ssize_t utf8nlen(utf8data_t, const char *, size_t);
+
+/*
+ * Cursor structure used by the normalizer.
+ */
+struct utf8cursor {
+	utf8data_t	data;
+	const char	*s;
+	const char	*p;
+	const char	*ss;
+	const char	*sp;
+	unsigned int	len;
+	unsigned int	slen;
+	short int	ccc;
+	short int	nccc;
+};
+
+/*
+ * Initialize a utf8cursor to normalize a string.
+ * Returns 0 on success.
+ * Returns -1 on failure.
+ */
+extern int utf8cursor(struct utf8cursor *, utf8data_t, const char *);
+extern int utf8ncursor(struct utf8cursor *, utf8data_t, const char *, size_t);
+
+/*
+ * Get the next byte in the normalization.
+ * Returns a value > 0 && < 256 on success.
+ * Returns 0 when the end of the normalization is reached.
+ * Returns -1 if the string being normalized is not valid UTF-8.
+ */
+extern int utf8byte(struct utf8cursor *);
+
+#endif /* UTF8NORM_H */
libxfs/Makefile +1 -0
index ae15a5d..a1e85ef 100644
@@ -14,6 +14,7 @@ HFILES = xfs.h init.h xfs_dir2_priv.h crc32defs.h crc32table.h
 CFILES = cache.c \
 	crc32.c \
 	init.c kmem.c logitem.c radix-tree.c rdwr.c trans.c util.c \
+	utf8norm.c \
 	xfs_alloc.c \
 	xfs_alloc_btree.c \
 	xfs_attr.c \
libxfs/utf8norm.c new file +643 -0

643 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#37 lore raw

PATCH 20/35 xfsprogs: reduce the size of utf8data[]

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Remove the Hangul decompositions from the utf8data trie, and do algorithmic decomposition to calculate them on the fly. To store the decomposition the caller of utf8lookup()/utf8nlookup() must provide a 12-byte buffer, which is used to synthesize a leaf with the decomposition. Trie size is reduced from 245kB to 90kB.

This change also contains a number of robustness fixes to the trie generator mkutf8data.c.

---
 include/utf8norm.h    |   4 +
 libxfs/utf8norm.c     | 190 ++++++++++++++++++++---
 utf8norm/mkutf8data.c | 421 ++++++++++++++++++++++++++++++++++++++------------
 3 files changed, 492 insertions(+), 123 deletions(-)
include/utf8norm.h +4 -0
index cd77580..44a869c 100644
@@ -22,6 +22,9 @@
 /* An opaque type used to determine the normalization in use. */
 typedef const struct utf8data *utf8data_t;
 
+/* Needed in struct utf8cursor below. */
+#define UTF8HANGULLEAF	(12)
+
 /* Encoding a unicode version number as a single unsigned int. */
 #define UNICODE_MAJ_SHIFT		(16)
 #define UNICODE_MIN_SHIFT		(8)
@@ -90,6 +93,7 @@ struct utf8cursor {
 	unsigned int	slen;
 	short int	ccc;
 	short int	nccc;
+	unsigned char	hangul[UTF8HANGULLEAF];
 };
 
 /*
libxfs/utf8norm.c +169 -21

315 lines not on this page · raw message

utf8norm/mkutf8data.c +319 -102

800 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#38 lore raw

PATCH 21/35 libxfs: return the first match during case-insensitive lookup

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.

---
 libxfs/xfs_dir2_block.c | 17 ++++-------
 libxfs/xfs_dir2_leaf.c  | 38 ++++-------------------
 libxfs/xfs_dir2_node.c  | 80 ++++++++++++++++++-------------------------------
 libxfs/xfs_dir2_sf.c    |  8 ++---
 4 files changed, 44 insertions(+), 99 deletions(-)
libxfs/xfs_dir2_block.c +5 -12
index cede01f..2880431 100644
@@ -705,28 +705,21 @@ xfs_dir2_block_lookup_int(
 		dep = (xfs_dir2_data_entry_t *)
 			((char *)hdr + xfs_dir2_dataptr_to_off(mp, addr));
 		/*
-		 * Compare name and if it's an exact match, return the index
-		 * and buffer. If it's the first case-insensitive match, store
-		 * the index and buffer and continue looking for an exact match.
+		 * Compare name and if it's a match, return the
+		 * index and buffer.
 		 */
 		cmp = mp->m_dirnameops->compname(args, dep->name, dep->namelen);
-		if (cmp != XFS_CMP_DIFFERENT && cmp != args->cmpresult) {
+		if (cmp != XFS_CMP_DIFFERENT) {
 			args->cmpresult = cmp;
 			*bpp = bp;
 			*entno = mid;
-			if (cmp == XFS_CMP_EXACT)
-				return 0;
+			return 0;
 		}
 	} while (++mid < be32_to_cpu(btp->count) &&
 			be32_to_cpu(blp[mid].hashval) == hash);
 
 	ASSERT(args->op_flags & XFS_DA_OP_OKNOENT);
-	/*
-	 * Here, we can only be doing a lookup (not a rename or replace).
-	 * If a case-insensitive match was found earlier, return success.
-	 */
-	if (args->cmpresult == XFS_CMP_CASE)
-		return 0;
+	ASSERT(args->cmpresult == XFS_CMP_DIFFERENT);
 	/*
 	 * No match, release the buffer and return ENOENT.
 	 */
libxfs/xfs_dir2_leaf.c +6 -32
index 8e0cbc9..b1901d3 100644
@@ -1246,7 +1246,6 @@ xfs_dir2_leaf_lookup_int(
 	xfs_mount_t		*mp;		/* filesystem mount point */
 	xfs_dir2_db_t		newdb;		/* new data block number */
 	xfs_trans_t		*tp;		/* transaction pointer */
-	xfs_dir2_db_t		cidb = -1;	/* case match data block no. */
 	enum xfs_dacmp		cmp;		/* name compare result */
 	struct xfs_dir2_leaf_entry *ents;
 	struct xfs_dir3_icleaf_hdr leafhdr;
@@ -1307,47 +1306,22 @@ xfs_dir2_leaf_lookup_int(
 		dep = (xfs_dir2_data_entry_t *)((char *)dbp->b_addr +
 			xfs_dir2_dataptr_to_off(mp, be32_to_cpu(lep->address)));
 		/*
-		 * Compare name and if it's an exact match, return the index
-		 * and buffer. If it's the first case-insensitive match, store
-		 * the index and buffer and continue looking for an exact match.
+		 * Compare name and if it's a match, return the index
+		 * and buffer.
 		 */
 		cmp = mp->m_dirnameops->compname(args, dep->name, dep->namelen);
-		if (cmp != XFS_CMP_DIFFERENT && cmp != args->cmpresult) {
+		if (cmp != XFS_CMP_DIFFERENT) {
 			args->cmpresult = cmp;
 			*indexp = index;
-			/* case exact match: return the current buffer. */
-			if (cmp == XFS_CMP_EXACT) {
-				*dbpp = dbp;
-				return 0;
-			}
-			cidb = curdb;
+			*dbpp = dbp;
+			return 0;
 		}
 	}
 	ASSERT(args->op_flags & XFS_DA_OP_OKNOENT);
-	/*
-	 * Here, we can only be doing a lookup (not a rename or remove).
-	 * If a case-insensitive match was found earlier, re-read the
-	 * appropriate data block if required and return it.
-	 */
-	if (args->cmpresult == XFS_CMP_CASE) {
-		ASSERT(cidb != -1);
-		if (cidb != curdb) {
-			xfs_trans_brelse(tp, dbp);
-			error = xfs_dir3_data_read(tp, dp,
-						   xfs_dir2_db_to_da(mp, cidb),
-						   -1, &dbp);
-			if (error) {
-				xfs_trans_brelse(tp, lbp);
-				return error;
-			}
-		}
-		*dbpp = dbp;
-		return 0;
-	}
+	ASSERT(args->cmpresult == XFS_CMP_DIFFERENT);
 	/*
 	 * No match found, return ENOENT.
 	 */
-	ASSERT(cidb == -1);
 	if (dbp)
 		xfs_trans_brelse(tp, dbp);
 	xfs_trans_brelse(tp, lbp);
libxfs/xfs_dir2_node.c +29 -51

121 lines not on this page · raw message

libxfs/xfs_dir2_sf.c +4 -4
index 7580333..7b01d43 100644
@@ -833,13 +833,12 @@ xfs_dir2_sf_lookup(
 	for (i = 0, sfep = xfs_dir2_sf_firstentry(sfp); i < sfp->count;
 	     i++, sfep = xfs_dir3_sf_nextentry(dp->i_mount, sfp, sfep)) {
 		/*
-		 * Compare name and if it's an exact match, return the inode
-		 * number. If it's the first case-insensitive match, store the
-		 * inode number and continue looking for an exact match.
+		 * Compare name and if it's a match, return the inode
+		 * number.
 		 */
 		cmp = dp->i_mount->m_dirnameops->compname(args, sfep->name,
 								sfep->namelen);
-		if (cmp != XFS_CMP_DIFFERENT && cmp != args->cmpresult) {
+		if (cmp != XFS_CMP_DIFFERENT) {
 			args->cmpresult = cmp;
 			args->inumber = xfs_dir3_sfe_get_ino(dp->i_mount,
 							     sfp, sfep);
@@ -848,6 +847,7 @@ xfs_dir2_sf_lookup(
 			if (cmp == XFS_CMP_EXACT)
 				return XFS_ERROR(EEXIST);
 			ci_sfep = sfep;
+			break;
 		}
 	}
 	ASSERT(args->op_flags & XFS_DA_OP_OKNOENT);
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#39 lore raw

PATCH 22/35 libxfs: rename XFS_CMP_CASE to XFS_CMP_MATCH

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Rename XFS_CMP_CASE to XFS_CMP_MATCH. With unicode filenames and normalization, different strings will match on other criteria than case insensitivity.

---
 include/xfs_da_btree.h | 2 +-
 libxfs/xfs_dir2.c      | 9 ++++++---
 libxfs/xfs_dir2_node.c | 2 +-
 3 files changed, 8 insertions(+), 5 deletions(-)
include/xfs_da_btree.h +1 -1
index e492dca..3d9f9dd 100644
@@ -34,7 +34,7 @@ struct zone;
 enum xfs_dacmp {
 	XFS_CMP_DIFFERENT,	/* names are completely different */
 	XFS_CMP_EXACT,		/* names are exactly the same */
-	XFS_CMP_CASE		/* names are same but differ in case */
+	XFS_CMP_MATCH		/* names are same but differ in encoding */
 };
 
 /*
libxfs/xfs_dir2.c +6 -3
index 4c8c836..57e98a3 100644
@@ -72,7 +72,7 @@ xfs_ascii_ci_compname(
 			continue;
 		if (tolower(args->name[i]) != tolower(name[i]))
 			return XFS_CMP_DIFFERENT;
-		result = XFS_CMP_CASE;
+		result = XFS_CMP_MATCH;
 	}
 
 	return result;
@@ -248,8 +248,11 @@ xfs_dir_cilookup_result(
 {
 	if (args->cmpresult == XFS_CMP_DIFFERENT)
 		return ENOENT;
-	if (args->cmpresult != XFS_CMP_CASE ||
-					!(args->op_flags & XFS_DA_OP_CILOOKUP))
+	if (args->cmpresult == XFS_CMP_EXACT)
+		return EEXIST;
+	ASSERT(args->cmpresult == XFS_CMP_MATCH);
+	/* Only dup the found name if XFS_DA_OP_CILOOKUP is set. */
+	if (!(args->op_flags & XFS_DA_OP_CILOOKUP))
 		return EEXIST;
 
 	args->value = kmem_alloc(len, KM_NOFS | KM_MAYFAIL);
libxfs/xfs_dir2_node.c +1 -1
index fb27506..550ca99 100644
@@ -2034,7 +2034,7 @@ xfs_dir2_node_lookup(
 	error = xfs_da3_node_lookup_int(state, &rval);
 	if (error)
 		rval = error;
-	else if (rval == ENOENT && args->cmpresult == XFS_CMP_CASE) {
+	else if (rval == ENOENT && args->cmpresult == XFS_CMP_MATCH) {
 		/* If a CI match, dup the actual name and return EEXIST */
 		xfs_dir2_data_entry_t	*dep;
 
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#40 lore raw

PATCH 23/35 libxfs: add xfs_nameops.normhash

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Add a normhash callout to the xfs_nameops. This callout takes an xfs_da_args structure as its argument, and calculates a hash value over the name. It may in the process create a normalized form of the name, and assign that to the norm/normlen fields in the xfs_da_args structure.

Changes: The pointer in kmem_free() was type converted to suppress compiler warnings.

---
 include/xfs_da_btree.h |  3 +++
 libxfs/xfs_da_btree.c  |  9 ++++++++
 libxfs/xfs_dir2.c      | 56 +++++++++++++++++++++++++++++++++++++++-----------
 3 files changed, 56 insertions(+), 12 deletions(-)
include/xfs_da_btree.h +3 -0
index 3d9f9dd..f532d63 100644
@@ -42,7 +42,9 @@ enum xfs_dacmp {
  */
 typedef struct xfs_da_args {
 	const __uint8_t	*name;		/* string (maybe not NULL terminated) */
+	const __uint8_t	*norm;		/* normalized name (may be NULL) */
 	int		namelen;	/* length of string (maybe no NULL) */
+	int		normlen;	/* length of normalized name */
 	__uint8_t	filetype;	/* filetype of inode for directories */
 	__uint8_t	*value;		/* set of bytes (maybe contain NULLs) */
 	int		valuelen;	/* length of value */
@@ -131,6 +133,7 @@ typedef struct xfs_da_state {
  */
 struct xfs_nameops {
 	xfs_dahash_t	(*hashname)(struct xfs_name *);
+	int		(*normhash)(struct xfs_da_args *);
 	enum xfs_dacmp	(*compname)(struct xfs_da_args *,
 					const unsigned char *, int);
 };
libxfs/xfs_da_btree.c +9 -0
index b731b54..eb97317 100644
@@ -2000,8 +2000,17 @@ xfs_default_hashname(
 	return xfs_da_hashname(name->name, name->len);
 }
 
+STATIC int
+xfs_da_normhash(
+	struct xfs_da_args *args)
+{
+	args->hashval = xfs_da_hashname(args->name, args->namelen);
+	return 0;
+}
+
 const struct xfs_nameops xfs_default_nameops = {
 	.hashname	= xfs_default_hashname,
+	.normhash	= xfs_da_normhash,
 	.compname	= xfs_da_compname
 };
 
libxfs/xfs_dir2.c +44 -12

162 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#41 lore raw

PATCH 24/35 libxfs: change interface of xfs_nameops.hashname

In reply to Ben Myers

From: Olaf Weber <[email protected]>

With the introduction of the xfs_nameops.normhash callout, all uses of the hashname callout now occur in places where an xfs_name structure must be explicitly created just to match the parameter passing convention of this callout. Change the arguments to a const unsigned char * and int instead.

[v2: pass a 3rd argument for sb_utf8version to hashname. --bpm]

---
 db/check.c              |  7 +++----
 include/xfs_da_btree.h  |  3 ++-
 libxfs/xfs_da_btree.c   | 18 ++++++++++--------
 libxfs/xfs_dir2.c       | 11 +++++++----
 libxfs/xfs_dir2_block.c |  7 +++----
 libxfs/xfs_dir2_data.c  |  7 +++----
 repair/phase6.c         |  3 ++-
 7 files changed, 30 insertions(+), 26 deletions(-)
db/check.c +3 -4
index 4fd9fd0..d317a71 100644
@@ -2212,7 +2212,6 @@ process_data_dir_v2(
 	int			stale = 0;
 	int			tag_err;
 	__be16			*tagp;
-	struct xfs_name		xname;
 
 	data = iocur_top->data;
 	block = iocur_top->data;
@@ -2323,9 +2322,9 @@ process_data_dir_v2(
 		tag_err += be16_to_cpu(*tagp) != (char *)dep - (char *)data;
 		addr = xfs_dir2_db_off_to_dataptr(mp, db,
 			(char *)dep - (char *)data);
-		xname.name = dep->name;
-		xname.len = dep->namelen;
-		dir_hash_add(mp->m_dirnameops->hashname(&xname), addr);
+		dir_hash_add(mp->m_dirnameops->hashname(dep->name,
+					dep->namelen,
+					0 /* version for later */), addr);
 		ptr += xfs_dir3_data_entsize(mp, dep->namelen);
 		count++;
 		lastfree = 0;
include/xfs_da_btree.h +2 -1
index f532d63..33efd3e 100644
@@ -132,7 +132,8 @@ typedef struct xfs_da_state {
  * Name ops for directory and/or attr name operations
  */
 struct xfs_nameops {
-	xfs_dahash_t	(*hashname)(struct xfs_name *);
+	xfs_dahash_t	(*hashname)(const unsigned char *, int,
+					unsigned int);
 	int		(*normhash)(struct xfs_da_args *);
 	enum xfs_dacmp	(*compname)(struct xfs_da_args *,
 					const unsigned char *, int);
libxfs/xfs_da_btree.c +10 -8
index eb97317..c9784bd 100644
@@ -1983,6 +1983,15 @@ xfs_da_hashname(const __uint8_t *name, int namelen)
 	}
 }
 
+xfs_dahash_t
+xfs_da_hashname_op(
+	const __uint8_t		*name,
+	int 			namelen,
+	unsigned int 		unused)
+{
+	return xfs_da_hashname(name, namelen);
+}
+
 enum xfs_dacmp
 xfs_da_compname(
 	struct xfs_da_args *args,
@@ -1993,13 +2002,6 @@ xfs_da_compname(
 					XFS_CMP_EXACT : XFS_CMP_DIFFERENT;
 }
 
-static xfs_dahash_t
-xfs_default_hashname(
-	struct xfs_name	*name)
-{
-	return xfs_da_hashname(name->name, name->len);
-}
-
 STATIC int
 xfs_da_normhash(
 	struct xfs_da_args *args)
@@ -2009,7 +2011,7 @@ xfs_da_normhash(
 }
 
 const struct xfs_nameops xfs_default_nameops = {
-	.hashname	= xfs_default_hashname,
+	.hashname	= xfs_da_hashname_op,
 	.normhash	= xfs_da_normhash,
 	.compname	= xfs_da_compname
 };
libxfs/xfs_dir2.c +7 -4
index e52d082..191925d 100644
@@ -43,13 +43,15 @@ const unsigned char xfs_mode_to_ftype[S_IFMT >> S_SHIFT] = {
  */
 STATIC xfs_dahash_t
 xfs_ascii_ci_hashname(
-	struct xfs_name	*name)
+	const unsigned char *name,
+	int len,
+	unsigned int unused)
 {
 	xfs_dahash_t	hash;
 	int		i;
 
-	for (i = 0, hash = 0; i < name->len; i++)
-		hash = tolower(name->name[i]) ^ rol32(hash, 7);
+	for (i = 0, hash = 0; i < len; i++)
+		hash = tolower(name[i]) ^ rol32(hash, 7);
 
 	return hash;
 }
@@ -475,7 +477,8 @@ xfs_dir_canenter(
 	args.name = name->name;
 	args.namelen = name->len;
 	args.filetype = name->type;
-	args.hashval = dp->i_mount->m_dirnameops->hashname(name);
+	args.hashval = dp->i_mount->m_dirnameops->hashname(name->name,
+			name->len, 0 /* version for later */);
 	args.dp = dp;
 	args.whichfork = XFS_DATA_FORK;
 	args.trans = tp;
libxfs/xfs_dir2_block.c +3 -4
index 2880431..c26308e 100644
@@ -1047,7 +1047,6 @@ xfs_dir2_sf_to_block(
 	xfs_dir2_sf_hdr_t	*sfp;		/* shortform header  */
 	__be16			*tagp;		/* end of data entry */
 	xfs_trans_t		*tp;		/* transaction pointer */
-	struct xfs_name		name;
 	struct xfs_ifork	*ifp;
 
 	trace_xfs_dir2_sf_to_block(args);
@@ -1205,10 +1204,10 @@ xfs_dir2_sf_to_block(
 		tagp = xfs_dir3_data_entry_tag_p(mp, dep);
 		*tagp = cpu_to_be16((char *)dep - (char *)hdr);
 		xfs_dir2_data_log_entry(tp, bp, dep);
-		name.name = sfep->name;
-		name.len = sfep->namelen;
 		blp[2 + i].hashval = cpu_to_be32(mp->m_dirnameops->
-							hashname(&name));
+					hashname(sfep->name,
+						 sfep->namelen,
+						 0 /* version for later */));
 		blp[2 + i].address = cpu_to_be32(xfs_dir2_byte_to_dataptr(mp,
 						 (char *)dep - (char *)hdr));
 		offset = (int)((char *)(tagp + 1) - (char *)hdr);
libxfs/xfs_dir2_data.c +3 -4
index dc9df4d..ada4d1d 100644
@@ -46,7 +46,6 @@ __xfs_dir3_data_check(
 	xfs_mount_t		*mp;		/* filesystem mount point */
 	char			*p;		/* current data position */
 	int			stale;		/* count of stale leaves */
-	struct xfs_name		name;
 
 	mp = bp->b_target->bt_mount;
 	hdr = bp->b_addr;
@@ -142,9 +141,9 @@ __xfs_dir3_data_check(
 			addr = xfs_dir2_db_off_to_dataptr(mp, mp->m_dirdatablk,
 				(xfs_dir2_data_aoff_t)
 				((char *)dep - (char *)hdr));
-			name.name = dep->name;
-			name.len = dep->namelen;
-			hash = mp->m_dirnameops->hashname(&name);
+			hash = mp->m_dirnameops->hashname(dep->name,
+					dep->namelen,
+					0 /* version for later */);
 			for (i = 0; i < be32_to_cpu(btp->count); i++) {
 				if (be32_to_cpu(lep[i].address) == addr &&
 				    be32_to_cpu(lep[i].hashval) == hash)
repair/phase6.c +2 -1
index f13069f..c18ef69 100644
@@ -195,7 +195,8 @@ dir_hash_add(
 	dup = 0;
 
 	if (!junk) {
-		hash = mp->m_dirnameops->hashname(&xname);
+		hash = mp->m_dirnameops->hashname(name, namelen,
+				0 /* version for later */);
 		byhash = DIR_HASH_FUNC(hashtab, hash);
 
 		/*
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#42 lore raw

PATCH 25/35 libxfs: add a superblock feature bit to indicate UTF-8 support.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

When UTF-8 support is enabled, the xfs_dir_ci_inode_operations must be installed. Add xfs_sb_version_hasci(), which tests both the borgbit and the utf8bit, and returns true if at least one of them is set. Replace calls to xfs_sb_version_hasasciici() as needed.

---
 include/xfs_fs.h |  1 +
 include/xfs_sb.h | 24 +++++++++++++++++++++++-
 2 files changed, 24 insertions(+), 1 deletion(-)
include/xfs_fs.h +1 -0
index 59c40fc..c260ec6 100644
@@ -239,6 +239,7 @@ typedef struct xfs_fsop_resblks {
 #define XFS_FSOP_GEOM_FLAGS_V5SB	0x8000	/* version 5 superblock */
 #define XFS_FSOP_GEOM_FLAGS_FTYPE	0x10000	/* inode directory types */
 #define XFS_FSOP_GEOM_FLAGS_FINOBT	0x20000	/* free inode btree */
+#define XFS_FSOP_GEOM_FLAGS_UTF8	0x40000	/* utf8 filenames */
 
 
 /*
include/xfs_sb.h +23 -1
index 950d1ea..c8563ce 100644
@@ -82,6 +82,7 @@ struct xfs_trans;
 #define XFS_SB_VERSION2_RESERVED4BIT	0x00000004
 #define XFS_SB_VERSION2_ATTR2BIT	0x00000008	/* Inline attr rework */
 #define XFS_SB_VERSION2_PARENTBIT	0x00000010	/* parent pointers */
+#define XFS_SB_VERSION2_UTF8BIT		0x00000020      /* utf8 names */
 #define XFS_SB_VERSION2_PROJID32BIT	0x00000080	/* 32 bit project id */
 #define XFS_SB_VERSION2_CRCBIT		0x00000100	/* metadata CRCs */
 #define XFS_SB_VERSION2_FTYPE		0x00000200	/* inode type in dir */
@@ -89,6 +90,7 @@ struct xfs_trans;
 #define	XFS_SB_VERSION2_OKREALFBITS	\
 	(XFS_SB_VERSION2_LAZYSBCOUNTBIT	| \
 	 XFS_SB_VERSION2_ATTR2BIT	| \
+	 XFS_SB_VERSION2_UTF8BIT	| \
 	 XFS_SB_VERSION2_PROJID32BIT	| \
 	 XFS_SB_VERSION2_FTYPE)
 #define	XFS_SB_VERSION2_OKSASHFBITS	\
@@ -600,8 +602,10 @@ xfs_sb_has_ro_compat_feature(
 }
 
 #define XFS_SB_FEAT_INCOMPAT_FTYPE	(1 << 0)	/* filetype in dirent */
+#define XFS_SB_FEAT_INCOMPAT_UTF8	(1 << 1)	/* utf-8 name support */
 #define XFS_SB_FEAT_INCOMPAT_ALL \
-		(XFS_SB_FEAT_INCOMPAT_FTYPE)
+		(XFS_SB_FEAT_INCOMPAT_FTYPE | \
+		 XFS_SB_FEAT_INCOMPAT_UTF8)
 
 #define XFS_SB_FEAT_INCOMPAT_UNKNOWN	~XFS_SB_FEAT_INCOMPAT_ALL
 static inline bool
@@ -649,6 +653,24 @@ static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp)
 		(sbp->sb_features_ro_compat & XFS_SB_FEAT_RO_COMPAT_FINOBT);
 }
 
+static inline int xfs_sb_version_hasutf8(xfs_sb_t *sbp)
+{
+	return (XFS_SB_VERSION_NUM(sbp) == XFS_SB_VERSION_5 &&
+		xfs_sb_has_incompat_feature(sbp, XFS_SB_FEAT_INCOMPAT_UTF8)) ||
+		(xfs_sb_version_hasmorebits(sbp) &&
+		(sbp->sb_features2 & XFS_SB_VERSION2_UTF8BIT));
+}
+
+/*
+ * Special case: there are a number of places where we need to test
+ * both the borgbit and the utf8bit, and take the same action if
+ * either of those is set.
+ */
+static inline int xfs_sb_version_hasci(xfs_sb_t *sbp)
+{
+	return xfs_sb_version_hasasciici(sbp) || xfs_sb_version_hasutf8(sbp);
+}
+
 /*
  * end of superblock version macros
  */
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#43 lore raw

PATCH 26/35 libxfs: store utf8version in the superblock

In reply to Ben Myers

From: Ben Myers <[email protected]>

The utf8 version a filesystem was created with needs to be stored in order that normalizations will remain stable over the lifetime of the filesystem. Convert sb_pad to sb_utf8version in the super block. This also adds checks at mount time to see whether the unicode normalization module has support for the version of unicode that the filesystem requires. If not we fail the mount.

---
 include/xfs_sb.h   | 10 ++++----
 include/xfs_utf8.h | 24 +++++++++++++++++++
 libxfs/xfs_sb.c    |  4 ++--
 libxfs/xfs_utf8.c  | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 102 insertions(+), 6 deletions(-)
 create mode 100644 include/xfs_utf8.h
 create mode 100644 libxfs/xfs_utf8.c
include/xfs_sb.h +6 -4
index c8563ce..a400cf1 100644
@@ -176,7 +176,7 @@ typedef struct xfs_sb {
 	__uint32_t	sb_features_log_incompat;
 
 	__uint32_t	sb_crc;		/* superblock crc */
-	__uint32_t	sb_pad;
+	__uint32_t	sb_utf8version;	/* unicode version */
 
 	xfs_ino_t	sb_pquotino;	/* project quota inode */
 	xfs_lsn_t	sb_lsn;		/* last write sequence */
@@ -262,7 +262,7 @@ typedef struct xfs_dsb {
 	__be32		sb_features_log_incompat;
 
 	__le32		sb_crc;		/* superblock crc */
-	__be32		sb_pad;
+	__be32		sb_utf8version;	/* version of unicode */
 
 	__be64		sb_pquotino;	/* project quota inode */
 	__be64		sb_lsn;		/* last write sequence */
@@ -288,7 +288,7 @@ typedef enum {
 	XFS_SBS_LOGSECTLOG, XFS_SBS_LOGSECTSIZE, XFS_SBS_LOGSUNIT,
 	XFS_SBS_FEATURES2, XFS_SBS_BAD_FEATURES2, XFS_SBS_FEATURES_COMPAT,
 	XFS_SBS_FEATURES_RO_COMPAT, XFS_SBS_FEATURES_INCOMPAT,
-	XFS_SBS_FEATURES_LOG_INCOMPAT, XFS_SBS_CRC, XFS_SBS_PAD,
+	XFS_SBS_FEATURES_LOG_INCOMPAT, XFS_SBS_CRC, XFS_SBS_UTF8VERSION,
 	XFS_SBS_PQUOTINO, XFS_SBS_LSN,
 	XFS_SBS_FIELDCOUNT
 } xfs_sb_field_t;
@@ -320,6 +320,7 @@ typedef enum {
 #define XFS_SB_FEATURES_INCOMPAT XFS_SB_MVAL(FEATURES_INCOMPAT)
 #define XFS_SB_FEATURES_LOG_INCOMPAT XFS_SB_MVAL(FEATURES_LOG_INCOMPAT)
 #define XFS_SB_CRC		XFS_SB_MVAL(CRC)
+#define XFS_SB_UTF8VERSION	XFS_SB_MVAL(UTF8VERSION)
 #define XFS_SB_PQUOTINO		XFS_SB_MVAL(PQUOTINO)
 #define	XFS_SB_NUM_BITS		((int)XFS_SBS_FIELDCOUNT)
 #define	XFS_SB_ALL_BITS		((1LL << XFS_SB_NUM_BITS) - 1)
@@ -330,7 +331,8 @@ typedef enum {
 	 XFS_SB_ICOUNT | XFS_SB_IFREE | XFS_SB_FDBLOCKS | XFS_SB_FEATURES2 | \
 	 XFS_SB_BAD_FEATURES2 | XFS_SB_FEATURES_COMPAT | \
 	 XFS_SB_FEATURES_RO_COMPAT | XFS_SB_FEATURES_INCOMPAT | \
-	 XFS_SB_FEATURES_LOG_INCOMPAT | XFS_SB_PQUOTINO)
+	 XFS_SB_FEATURES_LOG_INCOMPAT | XFS_SB_UTF8VERSION | \
+	 XFS_SB_PQUOTINO)
 
 
 /*
include/xfs_utf8.h new file +24 -0
new file mode 100644
index 0000000..8a700de
@@ -0,0 +1,24 @@
+/*
+ * Copyright (c) 2014 SGI.
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#ifndef XFS_UTF8_H
+#define XFS_UTF8_H
+
+extern int xfs_utf8_version_ok(struct xfs_mount *);
+
+#endif /* XFS_UTF8_H */
libxfs/xfs_sb.c +2 -2
index ea89367..642a2df 100644
@@ -78,7 +78,7 @@ static const struct {
 	{ offsetof(xfs_sb_t, sb_features_incompat),	0 },
 	{ offsetof(xfs_sb_t, sb_features_log_incompat),	0 },
 	{ offsetof(xfs_sb_t, sb_crc),		0 },
-	{ offsetof(xfs_sb_t, sb_pad),		0 },
+	{ offsetof(xfs_sb_t, sb_utf8version),	0 },
 	{ offsetof(xfs_sb_t, sb_pquotino),	0 },
 	{ offsetof(xfs_sb_t, sb_lsn),		0 },
 	{ sizeof(xfs_sb_t),			0 }
@@ -410,7 +410,7 @@ xfs_sb_from_disk(
 				be32_to_cpu(from->sb_features_log_incompat);
 	/* crc is only used on disk, not in memory; just init to 0 here. */
 	to->sb_crc = 0;
-	to->sb_pad = 0;
+	to->sb_utf8version = be32_to_cpu(from->sb_utf8version);
 	to->sb_pquotino = be64_to_cpu(from->sb_pquotino);
 	to->sb_lsn = be64_to_cpu(from->sb_lsn);
 }
libxfs/xfs_utf8.c new file +70 -0

70 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#44 lore raw

PATCH 27/35 libxfs: add xfs_nameops for utf8 and utf8+casefold.

In reply to Ben Myers

From: Olaf Weber <[email protected]>

The xfs_utf8_nameops use the nfkdi normalization when comparing filenames, and are installed if the utf8bit is set in the super block.

The xfs_utf8_ci_nameops use the nfkdicf normalization when comparing filenames, and are installed if both the utf8bit and the borgbit are set in the superblock.

Normalized filenames are not stored on disk. Normalization will fail if a filename is not valid UTF-8, in which case the filename is treated as an opaque blob.

[v2: updated to pass through sb_utf8version. -bpm]

---
 libxfs/xfs_utf8.c | 208 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)
libxfs/xfs_utf8.c +208 -0
index ebfdaec..3be1fbb 100644
@@ -68,3 +68,211 @@ xfs_utf8_version_ok(
 
 	return 0;
 }
+
+/*
+ * xfs nameops using nfkdi
+ */
+
+static xfs_dahash_t
+xfs_utf8_hashname(
+	const unsigned char *name,
+	int len,
+	unsigned int sb_utf8version)
+{
+	utf8data_t	nfkdi;
+	struct utf8cursor u8c;
+	xfs_dahash_t	hash;
+	int		val;
+
+	nfkdi = utf8nfkdi(sb_utf8version);
+	hash = 0;
+	if (utf8ncursor(&u8c, nfkdi, name, len) < 0)
+		goto blob;
+	while ((val = utf8byte(&u8c)) > 0)
+		hash = val ^ rol32(hash, 7);
+	/* In case of error treat the name as a binary blob. */
+	if (val == 0)
+		return hash;
+blob:
+	return xfs_da_hashname(name, len);
+}
+
+static int
+xfs_utf8_normhash(
+	struct xfs_da_args *args)
+{
+	utf8data_t	nfkdi;
+	struct utf8cursor u8c;
+	unsigned char	*norm;
+	ssize_t		normlen;
+	int		c;
+	unsigned int	sb_utf8version =
+		args->dp->i_mount->m_sb.sb_utf8version;
+
+	nfkdi = utf8nfkdi(sb_utf8version);
+	/* Failure to normalize is treated as a blob. */
+	if ((normlen = utf8nlen(nfkdi, args->name, args->namelen)) < 0)
+		goto blob;
+	if (utf8ncursor(&u8c, nfkdi, args->name, args->namelen) < 0)
+		goto blob;
+	if (!(norm = kmem_alloc(normlen + 1, KM_NOFS|KM_MAYFAIL)))
+		return -ENOMEM;
+	args->norm = norm;
+	args->normlen = normlen;
+	while ((c = utf8byte(&u8c)) > 0)
+		*norm++ = c;
+	if (c == 0) {
+		*norm = '\0';
+		args->hashval = xfs_da_hashname(args->norm, args->normlen);
+		return 0;
+	}
+	kmem_free(args->norm);
+blob:
+	args->norm = NULL;
+	args->normlen = -1;
+	args->hashval = xfs_da_hashname(args->name, args->namelen);
+	return 0;
+}
+
+static enum xfs_dacmp
+xfs_utf8_compname(
+	struct xfs_da_args *args,
+	const unsigned char *name,
+	int		len)
+{
+	utf8data_t	nfkdi;
+	struct utf8cursor u8c;
+	const unsigned char *norm;
+	int		c;
+	unsigned int	sb_utf8version =
+		args->dp->i_mount->m_sb.sb_utf8version;
+
+	ASSERT(args->norm || args->normlen == -1);
+
+	/* Check for an exact match first. */
+	if (args->namelen == len && memcmp(args->name, name, len) == 0)
+		return XFS_CMP_EXACT;
+	/* xfs_utf8_normhash() set args->normlen to -1 for a blob */
+	if (args->normlen < 0)
+		return XFS_CMP_DIFFERENT;
+	nfkdi = utf8nfkdi(sb_utf8version);
+	if (utf8ncursor(&u8c, nfkdi, name, len) < 0)
+		return XFS_CMP_DIFFERENT;
+	norm = args->norm;
+	while ((c = utf8byte(&u8c)) > 0)
+		if (c != *norm++)
+			return XFS_CMP_DIFFERENT;
+	if (c < 0 || *norm != '\0')
+		return XFS_CMP_DIFFERENT;
+	return XFS_CMP_MATCH;
+}
+
+struct xfs_nameops xfs_utf8_nameops = {
+	.hashname = xfs_utf8_hashname,
+	.normhash = xfs_utf8_normhash,
+	.compname = xfs_utf8_compname,
+};
+
+/*
+ * xfs nameops using nfkdicf
+ */
+
+static xfs_dahash_t
+xfs_utf8_ci_hashname(
+	const unsigned char *name,
+	int len,
+	unsigned int sb_utf8version)
+{
+	utf8data_t	nfkdicf;
+	struct utf8cursor u8c;
+	xfs_dahash_t	hash;
+	int		val;
+
+	nfkdicf = utf8nfkdicf(sb_utf8version);
+	hash = 0;
+	if (utf8ncursor(&u8c, nfkdicf, name, len) < 0)
+		goto blob;
+	while ((val = utf8byte(&u8c)) > 0)
+		hash = val ^ rol32(hash, 7);
+	/* In case of error treat the name as a binary blob. */
+	if (val == 0)
+		return hash;
+blob:
+	return xfs_da_hashname(name, len);
+}
+
+static int
+xfs_utf8_ci_normhash(
+	struct xfs_da_args *args)
+{
+	utf8data_t	nfkdicf;
+	struct utf8cursor u8c;
+	unsigned char	*norm;
+	ssize_t		normlen;
+	int		c;
+	unsigned int	sb_utf8version =
+		args->dp->i_mount->m_sb.sb_utf8version;
+
+	nfkdicf = utf8nfkdicf(sb_utf8version);
+	/* Failure to normalize is treated as a blob. */
+	if ((normlen = utf8nlen(nfkdicf, args->name, args->namelen)) < 0)
+		goto blob;
+	if (utf8ncursor(&u8c, nfkdicf, args->name, args->namelen) < 0)
+		goto blob;
+	if (!(norm = kmem_alloc(normlen + 1, KM_NOFS|KM_MAYFAIL)))
+		return -ENOMEM;
+	args->norm = norm;
+	args->normlen = normlen;
+	while ((c = utf8byte(&u8c)) > 0)
+		*norm++ = c;
+	if (c == 0) {
+		*norm = '\0';
+		args->hashval = xfs_da_hashname(args->norm, args->normlen);
+		return 0;
+	}
+	kmem_free(args->norm);
+blob:
+	args->norm = NULL;
+	args->normlen = -1;
+	args->hashval = xfs_da_hashname(args->name, args->namelen);
+	return 0;
+}
+
+static enum xfs_dacmp
+xfs_utf8_ci_compname(
+	struct xfs_da_args *args,
+	const unsigned char *name,
+	int		len)
+{
+	utf8data_t	nfkdicf;
+	struct utf8cursor u8c;
+	const unsigned char *norm;
+	int		c;
+	unsigned int	sb_utf8version =
+		args->dp->i_mount->m_sb.sb_utf8version;
+
+	ASSERT(args->norm || args->normlen == -1);
+
+	/* Check for an exact match first. */
+	if (args->namelen == len && memcmp(args->name, name, len) == 0)
+		return XFS_CMP_EXACT;
+	/* xfs_utf8_ci_normhash() set args->normlen to -1 for a blob */
+	if (args->normlen < 0)
+		return XFS_CMP_DIFFERENT;
+	nfkdicf = utf8nfkdicf(sb_utf8version);
+	if (utf8ncursor(&u8c, nfkdicf, name, len) < 0)
+		return XFS_CMP_DIFFERENT;
+	norm = args->norm;
+	while ((c = utf8byte(&u8c)) > 0)
+		if (c != *norm++)
+			return XFS_CMP_DIFFERENT;
+	if (c < 0 || *norm != '\0')
+		return XFS_CMP_DIFFERENT;
+	return XFS_CMP_MATCH;
+}
+
+struct xfs_nameops xfs_utf8_ci_nameops = {
+	.hashname = xfs_utf8_ci_hashname,
+	.normhash = xfs_utf8_ci_normhash,
+	.compname = xfs_utf8_ci_compname,
+};
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#45 lore raw

PATCH 28/35 libxfs: apply utf-8 normalization rules to user extended attribute names

In reply to Ben Myers

From: Olaf Weber <[email protected]>

Apply the same rules for UTF-8 normalization to the names of user-defined extended attributes. System attributes are excluded because they are not user-visible in the first place, and the kernel is expected to know what it is doing when naming them.

---
 libxfs/xfs_attr.c      | 49 +++++++++++++++++++++++++++++++++++++++++--------
 libxfs/xfs_attr_leaf.c | 11 +++++++++--
 libxfs/xfs_utf8.c      |  6 ++++++
 3 files changed, 56 insertions(+), 10 deletions(-)
libxfs/xfs_attr.c +41 -8
index 17519d3..c30703b 100644
@@ -88,8 +88,9 @@ xfs_attr_get_int(
 	int			*valuelenp,
 	int			flags)
 {
-	xfs_da_args_t   args;
-	int             error;
+	xfs_da_args_t   	args;
+	struct xfs_mount	*mp = ip->i_mount;
+	int             	error;
 
 	if (!xfs_inode_hasattr(ip))
 		return ENOATTR;
@@ -103,9 +104,12 @@ xfs_attr_get_int(
 	args.value = value;
 	args.valuelen = *valuelenp;
 	args.flags = flags;
-	args.hashval = xfs_da_hashname(args.name, args.namelen);
 	args.dp = ip;
 	args.whichfork = XFS_ATTR_FORK;
+	if (! xfs_sb_version_hasutf8(&mp->m_sb))
+		args.hashval = xfs_da_hashname(args.name, args.namelen);
+	else if ((error = mp->m_dirnameops->normhash(&args)) != 0)
+		return error;
 
 	/*
 	 * Decide on what work routines to call based on the inode size.
@@ -118,6 +122,9 @@ xfs_attr_get_int(
 		error = xfs_attr_node_get(&args);
 	}
 
+	if (args.norm)
+		kmem_free((void *)args.norm);
+
 	/*
 	 * Return the number of bytes in the value to the caller.
 	 */
@@ -239,12 +246,15 @@ xfs_attr_set_int(
 	args.value = value;
 	args.valuelen = valuelen;
 	args.flags = flags;
-	args.hashval = xfs_da_hashname(args.name, args.namelen);
 	args.dp = dp;
 	args.firstblock = &firstblock;
 	args.flist = &flist;
 	args.whichfork = XFS_ATTR_FORK;
 	args.op_flags = XFS_DA_OP_ADDNAME | XFS_DA_OP_OKNOENT;
+	if (! xfs_sb_version_hasutf8(&mp->m_sb))
+		args.hashval = xfs_da_hashname(args.name, args.namelen);
+	else if ((error = mp->m_dirnameops->normhash(&args)) != 0)
+		return error;
 
 	/* Size is now blocks for attribute data */
 	args.total = xfs_attr_calc_size(dp, name->len, valuelen, &local);
@@ -276,6 +286,8 @@ xfs_attr_set_int(
 	error = xfs_trans_reserve(args.trans, &tres, args.total, 0);
 	if (error) {
 		xfs_trans_cancel(args.trans, 0);
+		if (args.norm)
+			kmem_free((void *)args.norm);
 		return(error);
 	}
 	xfs_ilock(dp, XFS_ILOCK_EXCL);
@@ -286,6 +298,8 @@ xfs_attr_set_int(
 	if (error) {
 		xfs_iunlock(dp, XFS_ILOCK_EXCL);
 		xfs_trans_cancel(args.trans, XFS_TRANS_RELEASE_LOG_RES);
+		if (args.norm)
+			kmem_free((void *)args.norm);
 		return (error);
 	}
 
@@ -333,7 +347,8 @@ xfs_attr_set_int(
 			err2 = xfs_trans_commit(args.trans,
 						 XFS_TRANS_RELEASE_LOG_RES);
 			xfs_iunlock(dp, XFS_ILOCK_EXCL);
-
+			if (args.norm)
+				kmem_free((void *)args.norm);
 			return(error == 0 ? err2 : error);
 		}
 
@@ -398,6 +413,8 @@ xfs_attr_set_int(
 	xfs_trans_log_inode(args.trans, dp, XFS_ILOG_CORE);
 	error = xfs_trans_commit(args.trans, XFS_TRANS_RELEASE_LOG_RES);
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free((void *)args.norm);
 
 	return(error);
 
@@ -406,6 +423,9 @@ out:
 		xfs_trans_cancel(args.trans,
 			XFS_TRANS_RELEASE_LOG_RES|XFS_TRANS_ABORT);
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free((void *)args.norm);
+
 	return(error);
 }
 
@@ -452,12 +472,15 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags)
 	args.name = name->name;
 	args.namelen = name->len;
 	args.flags = flags;
-	args.hashval = xfs_da_hashname(args.name, args.namelen);
 	args.dp = dp;
 	args.firstblock = &firstblock;
 	args.flist = &flist;
 	args.total = 0;
 	args.whichfork = XFS_ATTR_FORK;
+	if (! xfs_sb_version_hasutf8(&mp->m_sb))
+		args.hashval = xfs_da_hashname(args.name, args.namelen);
+	else if ((error = mp->m_dirnameops->normhash(&args)) != 0)
+		return error;
 
 	/*
 	 * we have no control over the attribute names that userspace passes us
@@ -470,8 +493,11 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags)
 	 * Attach the dquots to the inode.
 	 */
 	error = xfs_qm_dqattach(dp, 0);
-	if (error)
-		return error;
+	if (error) {
+		if (args.norm)
+			kmem_free((void *)args.norm);
+			return error;
+	}
 
 	/*
 	 * Start our first transaction of the day.
@@ -497,6 +523,8 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags)
 				  XFS_ATTRRM_SPACE_RES(mp), 0);
 	if (error) {
 		xfs_trans_cancel(args.trans, 0);
+		if (args.norm)
+			kmem_free((void *)args.norm);
 		return(error);
 	}
 
@@ -546,6 +574,8 @@ xfs_attr_remove_int(xfs_inode_t *dp, struct xfs_name *name, int flags)
 	xfs_trans_log_inode(args.trans, dp, XFS_ILOG_CORE);
 	error = xfs_trans_commit(args.trans, XFS_TRANS_RELEASE_LOG_RES);
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free((void *)args.norm);
 
 	return(error);
 
@@ -554,6 +584,9 @@ out:
 		xfs_trans_cancel(args.trans,
 			XFS_TRANS_RELEASE_LOG_RES|XFS_TRANS_ABORT);
 	xfs_iunlock(dp, XFS_ILOCK_EXCL);
+	if (args.norm)
+		kmem_free((void *)args.norm);
+
 	return(error);
 }
 
libxfs/xfs_attr_leaf.c +9 -2
index f7f02ae..052a6a1 100644
@@ -634,6 +634,7 @@ int
 xfs_attr_shortform_to_leaf(xfs_da_args_t *args)
 {
 	xfs_inode_t *dp;
+	struct xfs_mount *mp;
 	xfs_attr_shortform_t *sf;
 	xfs_attr_sf_entry_t *sfe;
 	xfs_da_args_t nargs;
@@ -646,6 +647,7 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args)
 	trace_xfs_attr_sf_to_leaf(args);
 
 	dp = args->dp;
+	mp = dp->i_mount;
 	ifp = dp->i_afp;
 	sf = (xfs_attr_shortform_t *)ifp->if_u1.if_data;
 	size = be16_to_cpu(sf->hdr.totsize);
@@ -698,13 +700,18 @@ xfs_attr_shortform_to_leaf(xfs_da_args_t *args)
 		nargs.namelen = sfe->namelen;
 		nargs.value = &sfe->nameval[nargs.namelen];
 		nargs.valuelen = sfe->valuelen;
-		nargs.hashval = xfs_da_hashname(sfe->nameval,
-						sfe->namelen);
 		nargs.flags = XFS_ATTR_NSP_ONDISK_TO_ARGS(sfe->flags);
+		if (! xfs_sb_version_hasutf8(&mp->m_sb))
+			nargs.hashval = xfs_da_hashname(sfe->nameval,
+							sfe->namelen);
+		else if ((error = mp->m_dirnameops->normhash(&nargs)) != 0)
+			goto out;
 		error = xfs_attr3_leaf_lookup_int(bp, &nargs); /* set a->index */
 		ASSERT(error == ENOATTR);
 		error = xfs_attr3_leaf_add(bp, &nargs);
 		ASSERT(error != ENOSPC);
+		if (nargs.norm)
+			 kmem_free((void *)nargs.norm);
 		if (error)
 			goto out;
 		sfe = XFS_ATTR_SF_NEXTENTRY(sfe);
libxfs/xfs_utf8.c +6 -0
index 3be1fbb..f7042ef 100644
@@ -109,6 +109,9 @@ xfs_utf8_normhash(
 	unsigned int	sb_utf8version =
 		args->dp->i_mount->m_sb.sb_utf8version;
 
+	/* Don't normalize system attribute names. */
+	if (args->flags & (ATTR_ROOT|ATTR_SECURE))
+		goto blob;
 	nfkdi = utf8nfkdi(sb_utf8version);
 	/* Failure to normalize is treated as a blob. */
 	if ((normlen = utf8nlen(nfkdi, args->name, args->namelen)) < 0)
@@ -213,6 +216,9 @@ xfs_utf8_ci_normhash(
 	unsigned int	sb_utf8version =
 		args->dp->i_mount->m_sb.sb_utf8version;
 
+	/* Don't normalize system attribute names. */
+	if (args->flags & (ATTR_ROOT|ATTR_SECURE))
+		goto blob;
 	nfkdicf = utf8nfkdicf(sb_utf8version);
 	/* Failure to normalize is treated as a blob. */
 	if ((normlen = utf8nlen(nfkdicf, args->name, args->namelen)) < 0)
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#46 lore raw

PATCH 29/35 libxfs: rename XFS_IOC_FSGEOM to XFS_IOC_FSGEOM_V2

In reply to Ben Myers

From: Ben Myers <[email protected]>

We'll be creating a new versioned XFS_IOC_FSGEOMETRY ioctl and structure so rename the current revision to _V2.

---
 growfs/xfs_growfs.c |  8 ++++----
 include/xfs_fs.h    |  8 ++++----
 io/bmap.c           |  2 +-
 io/init.c           |  2 +-
 io/io.h             |  6 +++---
 io/open.c           | 10 +++++-----
 quota/free.c        |  2 +-
 7 files changed, 19 insertions(+), 19 deletions(-)
growfs/xfs_growfs.c +4 -4
index 8e611b6..c3df0c0 100644
@@ -44,7 +44,7 @@ Options:\n\
 
 void
 report_info(
-	xfs_fsop_geom_t	geo,
+	xfs_fsop_geom_v2_t	geo,
 	char		*mntpoint,
 	int		isint,
 	char		*logname,
@@ -101,7 +101,7 @@ main(int argc, char **argv)
 	int			error;	/* we have hit an error */
 	long			esize;	/* new rt extent size */
 	int			ffd;	/* mount point file descriptor */
-	xfs_fsop_geom_t		geo;	/* current fs geometry */
+	xfs_fsop_geom_v2_t	geo;	/* current fs geometry */
 	int			iflag;	/* -i flag */
 	int			isint;	/* log is currently internal */
 	int			lflag;	/* -l flag */
@@ -109,7 +109,7 @@ main(int argc, char **argv)
 	int			maxpct;	/* -m flag value */
 	int			mflag;	/* -m flag */
 	int			nflag;	/* -n flag */
-	xfs_fsop_geom_t		ngeo;	/* new fs geometry */
+	xfs_fsop_geom_v2_t	ngeo;	/* new fs geometry */
 	int			rflag;	/* -r flag */
 	long long		rsize;	/* new rt size in fs blocks */
 	int			ci;	/* ASCII case-insensitive fs */
@@ -220,7 +220,7 @@ main(int argc, char **argv)
 	}
 
 	/* get the current filesystem size & geometry */
-	if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY, &geo) < 0) {
+	if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY_V2, &geo) < 0) {
 		/*
 		 * OK, new xfsctl barfed - back off and try earlier version
 		 * as we're probably running an older kernel version.
include/xfs_fs.h +4 -4
index c260ec6..38faf5d 100644
@@ -180,9 +180,9 @@ typedef struct xfs_fsop_geom_v1 {
 } xfs_fsop_geom_v1_t;
 
 /*
- * Output for XFS_IOC_FSGEOMETRY
+ * Output for XFS_IOC_FSGEOMETRY_V2
  */
-typedef struct xfs_fsop_geom {
+typedef struct xfs_fsop_geom_v2 {
 	__u32		blocksize;	/* filesystem (data) block size */
 	__u32		rtextsize;	/* realtime extent size		*/
 	__u32		agblocks;	/* fsblocks in an AG		*/
@@ -204,7 +204,7 @@ typedef struct xfs_fsop_geom {
 	__u32		rtsectsize;	/* realtime sector size, bytes	*/
 	__u32		dirblocksize;	/* directory block size, bytes	*/
 	__u32		logsunit;	/* log stripe unit, bytes */
-} xfs_fsop_geom_t;
+} xfs_fsop_geom_v2_t;
 
 /* Output for XFS_FS_COUNTS */
 typedef struct xfs_fsop_counts {
@@ -553,7 +553,7 @@ typedef struct xfs_swapext
 #define XFS_IOC_FSSETDM_BY_HANDLE    _IOW ('X', 121, struct xfs_fsop_setdm_handlereq)
 #define XFS_IOC_ATTRLIST_BY_HANDLE   _IOW ('X', 122, struct xfs_fsop_attrlist_handlereq)
 #define XFS_IOC_ATTRMULTI_BY_HANDLE  _IOW ('X', 123, struct xfs_fsop_attrmulti_handlereq)
-#define XFS_IOC_FSGEOMETRY	     _IOR ('X', 124, struct xfs_fsop_geom)
+#define XFS_IOC_FSGEOMETRY_V2	     _IOR ('X', 124, struct xfs_fsop_geom_v2)
 #define XFS_IOC_GOINGDOWN	     _IOR ('X', 125, __uint32_t)
 /*	XFS_IOC_GETFSUUID ---------- deprecated 140	 */
 
io/bmap.c +1 -1
index a78cbb1..614eba1 100644
@@ -70,7 +70,7 @@ bmap_f(
 {
 	struct fsxattr		fsx;
 	struct getbmapx		*map;
-	struct xfs_fsop_geom	fsgeo;
+	struct xfs_fsop_geom_v2	fsgeo;
 	int			map_size;
 	int			loop = 0;
 	int			flg = 0;
io/init.c +1 -1
index bfc35bf..9622a6c 100644
@@ -127,7 +127,7 @@ init(
 	int		c, flags = 0;
 	char		*sp;
 	mode_t		mode = 0600;
-	xfs_fsop_geom_t	geometry = { 0 };
+	xfs_fsop_geom_v2_t	geometry = { 0 };
 
 	progname = basename(argv[0]);
 	setlocale(LC_ALL, "");
io/io.h +3 -3
index 1b3bca1..1837fe4 100644
@@ -44,7 +44,7 @@ typedef struct fileio {
 	int		fd;		/* open file descriptor */
 	int		flags;		/* flags describing file state */
 	char		*name;		/* file name at time of open */
-	xfs_fsop_geom_t	geom;		/* XFS filesystem geometry */
+	xfs_fsop_geom_v2_t	geom;	/* XFS filesystem geometry */
 } fileio_t;
 
 extern fileio_t		*filetable;	/* open file table */
@@ -74,8 +74,8 @@ extern void *check_mapping_range(mmap_region_t *, off64_t, size_t, int);
  */
 
 extern off64_t		filesize(void);
-extern int		openfile(char *, xfs_fsop_geom_t *, int, mode_t);
-extern int		addfile(char *, int , xfs_fsop_geom_t *, int);
+extern int		openfile(char *, xfs_fsop_geom_v2_t *, int, mode_t);
+extern int		addfile(char *, int , xfs_fsop_geom_v2_t *, int);
 extern void		printxattr(uint, int, int, const char *, int, int);
 
 extern unsigned int	recurse_all;
io/open.c +5 -5
index c106fa7..81c19c9 100644
@@ -140,7 +140,7 @@ stat_f(
 int
 openfile(
 	char		*path,
-	xfs_fsop_geom_t	*geom,
+	xfs_fsop_geom_v2_t	*geom,
 	int		flags,
 	mode_t		mode)
 {
@@ -185,7 +185,7 @@ openfile(
 	if (!geom || !platform_test_xfs_fd(fd))
 		return fd;
 
-	if (xfsctl(path, fd, XFS_IOC_FSGEOMETRY, geom) < 0) {
+	if (xfsctl(path, fd, XFS_IOC_FSGEOMETRY_V2, geom) < 0) {
 		perror("XFS_IOC_FSGEOMETRY");
 		close(fd);
 		return -1;
@@ -215,7 +215,7 @@ int
 addfile(
 	char		*name,
 	int		fd,
-	xfs_fsop_geom_t	*geometry,
+	xfs_fsop_geom_v2_t	*geometry,
 	int		flags)
 {
 	char		*filename;
@@ -284,7 +284,7 @@ open_f(
 	int		c, fd, flags = 0;
 	char		*sp;
 	mode_t		mode = 0600;
-	xfs_fsop_geom_t	geometry = { 0 };
+	xfs_fsop_geom_v2_t	geometry = { 0 };
 
 	if (argc == 1) {
 		if (file)
@@ -701,7 +701,7 @@ statfs_f(
 	char			**argv)
 {
 	struct xfs_fsop_counts	fscounts;
-	struct xfs_fsop_geom	fsgeo;
+	struct xfs_fsop_geom_v2	fsgeo;
 	struct statfs		st;
 
 	printf(_("fd.path = \"%s\"\n"), file->name);
quota/free.c +1 -1
index 79b52e9..b2e325b 100644
@@ -59,7 +59,7 @@ mount_free_space_data(
 	__uint64_t		*rfree)
 {
 	struct xfs_fsop_counts	fscounts;
-	struct xfs_fsop_geom	fsgeo;
+	struct xfs_fsop_geom_v2	fsgeo;
 	struct statfs		st;
 	__uint64_t		logsize, count, free;
 	int			fd;
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#47 lore raw

PATCH 30/35 libxfs: add versioned fsgeom ioctl with utf8version field

In reply to Ben Myers

From: Ben Myers <[email protected]>

This adds a utf8version field to the xfs_fs_geom structure. An important characteristic of this version of the ioctl is that fsgeo.version needs to be set by the caller to specify which version of the structure to return.

---
 include/xfs_fs.h | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
include/xfs_fs.h +31 -0
index 38faf5d..f920498 100644
@@ -206,6 +206,34 @@ typedef struct xfs_fsop_geom_v2 {
 	__u32		logsunit;	/* log stripe unit, bytes */
 } xfs_fsop_geom_v2_t;
 
+/*
+ * Output for XFS_IOC_FSGEOMETRY
+ */
+typedef struct xfs_fsop_geom {
+	__u32		blocksize;	/* filesystem (data) block size */
+	__u32		rtextsize;	/* realtime extent size		*/
+	__u32		agblocks;	/* fsblocks in an AG		*/
+	__u32		agcount;	/* number of allocation groups	*/
+	__u32		logblocks;	/* fsblocks in the log		*/
+	__u32		sectsize;	/* (data) sector size, bytes	*/
+	__u32		inodesize;	/* inode size in bytes		*/
+	__u32		imaxpct;	/* max allowed inode space(%)	*/
+	__u64		datablocks;	/* fsblocks in data subvolume	*/
+	__u64		rtblocks;	/* fsblocks in realtime subvol	*/
+	__u64		rtextents;	/* rt extents in realtime subvol*/
+	__u64		logstart;	/* starting fsblock of the log	*/
+	unsigned char	uuid[16];	/* unique id of the filesystem	*/
+	__u32		sunit;		/* stripe unit, fsblocks	*/
+	__u32		swidth;		/* stripe width, fsblocks	*/
+	__s32		version;	/* structure version		*/
+	__u32		flags;		/* superblock version flags	*/
+	__u32		logsectsize;	/* log sector size, bytes	*/
+	__u32		rtsectsize;	/* realtime sector size, bytes	*/
+	__u32		dirblocksize;	/* directory block size, bytes	*/
+	__u32		logsunit;	/* log stripe unit, bytes */
+	__u32		utf8version;	/* Unicode version		*/
+} xfs_fsop_geom_t;
+
 /* Output for XFS_FS_COUNTS */
 typedef struct xfs_fsop_counts {
 	__u64	freedata;	/* free data section blocks */
@@ -221,6 +249,8 @@ typedef struct xfs_fsop_resblks {
 } xfs_fsop_resblks_t;
 
 #define XFS_FSOP_GEOM_VERSION	0
+/* skipped 1-4 to match existing new_version xfs_fs_geometry argument */
+#define XFS_FSOP_GEOM_VERSION5	5
 
 #define XFS_FSOP_GEOM_FLAGS_ATTR	0x0001	/* attributes in use	*/
 #define XFS_FSOP_GEOM_FLAGS_NLINK	0x0002	/* 32-bit nlink values	*/
@@ -555,6 +585,7 @@ typedef struct xfs_swapext
 #define XFS_IOC_ATTRMULTI_BY_HANDLE  _IOW ('X', 123, struct xfs_fsop_attrmulti_handlereq)
 #define XFS_IOC_FSGEOMETRY_V2	     _IOR ('X', 124, struct xfs_fsop_geom_v2)
 #define XFS_IOC_GOINGDOWN	     _IOR ('X', 125, __uint32_t)
+#define XFS_IOC_FSGEOMETRY	     _IOR ('X', 126, struct xfs_fsop_geom)
 /*	XFS_IOC_GETFSUUID ---------- deprecated 140	 */
 
 
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#48 lore raw

PATCH 31/35 xfsprogs: add utf8 support to growfs

In reply to Ben Myers

From: Mark Tinguely <[email protected]>

Add utf-8 to xfs_growfs and xfs_info.

[v2: use versioned fsgeom ioctl. -bpm]

---
 growfs/xfs_growfs.c | 85 ++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 61 insertions(+), 24 deletions(-)
growfs/xfs_growfs.c +61 -24
index c3df0c0..5e9d575 100644
@@ -18,6 +18,7 @@
 
 #include <xfs/libxfs.h>
 #include <xfs/path.h>
+#include <xfs/utf8norm.h>
 
 static void
 usage(void)
@@ -44,7 +45,8 @@ Options:\n\
 
 void
 report_info(
-	xfs_fsop_geom_v2_t	geo,
+	xfs_fsop_geom_t	geo,
+	int		oldioctl,
 	char		*mntpoint,
 	int		isint,
 	char		*logname,
@@ -57,8 +59,31 @@ report_info(
 	int		crcs_enabled,
 	int		cimode,
 	int		ftype_enabled,
-	int		finobt_enabled)
+	int		finobt_enabled,
+	int		utf8)
 {
+	char		utf8_version_string[10];
+
+	/* XXX Can we assume that geo.version has always been zeroed by
+	 * the kernel so it is always meaningful? */
+	if (!oldioctl && geo.version >= XFS_FSOP_GEOM_VERSION5 && utf8) {
+		int	major, minor, revision;
+
+		major = geo.utf8version >> UNICODE_MAJ_SHIFT;
+		minor = (geo.utf8version & 0xff00) >> UNICODE_MIN_SHIFT;
+		revision = geo.utf8version & 0xff;
+
+		if (!revision && !minor)
+			sprintf(utf8_version_string, "%d", major);
+		else if (!revision)
+			sprintf(utf8_version_string, "%d.%d", major, minor);
+		else
+			sprintf(utf8_version_string, "%d.%d.%d",
+					major, minor, revision);
+	} else {
+		sprintf(utf8_version_string, "0");
+	}
+
 	printf(_(
 	    "meta-data=%-22s isize=%-6u agcount=%u, agsize=%u blks\n"
 	    "         =%-22s sectsz=%-5u attr=%u, projid32bit=%u\n"
@@ -66,6 +91,7 @@ report_info(
 	    "data     =%-22s bsize=%-6u blocks=%llu, imaxpct=%u\n"
 	    "         =%-22s sunit=%-6u swidth=%u blks\n"
 	    "naming   =version %-14u bsize=%-6u ascii-ci=%d ftype=%d\n"
+	    "         =%-22s utf8=%s\n"
 	    "log      =%-22s bsize=%-6u blocks=%u, version=%u\n"
 	    "         =%-22s sectsz=%-5u sunit=%u blks, lazy-count=%u\n"
 	    "realtime =%-22s extsz=%-6u blocks=%llu, rtextents=%llu\n"),
@@ -76,7 +102,8 @@ report_info(
 		"", geo.blocksize, (unsigned long long)geo.datablocks,
 			geo.imaxpct,
 		"", geo.sunit, geo.swidth,
-  		dirversion, geo.dirblocksize, cimode, ftype_enabled,
+		dirversion, geo.dirblocksize, cimode, ftype_enabled,
+		"", utf8_version_string,
 		isint ? _("internal") : logname ? logname : _("external"),
 			geo.blocksize, geo.logblocks, logversion,
 		"", geo.logsectsize, geo.logsunit / geo.blocksize, lazycount,
@@ -101,7 +128,7 @@ main(int argc, char **argv)
 	int			error;	/* we have hit an error */
 	long			esize;	/* new rt extent size */
 	int			ffd;	/* mount point file descriptor */
-	xfs_fsop_geom_v2_t	geo;	/* current fs geometry */
+	xfs_fsop_geom_t		geo;	/* current fs geometry */
 	int			iflag;	/* -i flag */
 	int			isint;	/* log is currently internal */
 	int			lflag;	/* -l flag */
@@ -109,11 +136,12 @@ main(int argc, char **argv)
 	int			maxpct;	/* -m flag value */
 	int			mflag;	/* -m flag */
 	int			nflag;	/* -n flag */
-	xfs_fsop_geom_v2_t	ngeo;	/* new fs geometry */
+	xfs_fsop_geom_t		ngeo;	/* new fs geometry */
 	int			rflag;	/* -r flag */
 	long long		rsize;	/* new rt size in fs blocks */
 	int			ci;	/* ASCII case-insensitive fs */
 	int			lazycount; /* lazy superblock counters */
+	int			utf8;	/* Unicode chars supported */
 	int			xflag;	/* -x flag */
 	char			*fname;	/* mount point name */
 	char			*datadev; /* data device name */
@@ -125,6 +153,7 @@ main(int argc, char **argv)
 	int			crcs_enabled;
 	int			ftype_enabled = 0;
 	int			finobt_enabled;	/* free inode btree */
+	int			oldioctl = 0;
 
 	progname = basename(argv[0]);
 	setlocale(LC_ALL, "");
@@ -219,21 +248,28 @@ main(int argc, char **argv)
 		exit(1);
 	}
 
-	/* get the current filesystem size & geometry */
-	if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY_V2, &geo) < 0) {
-		/*
-		 * OK, new xfsctl barfed - back off and try earlier version
-		 * as we're probably running an older kernel version.
-		 * Only field added in the v2 geometry xfsctl is "logsunit"
-		 * so we'll zero that out for later display (as zero).
-		 */
-		geo.logsunit = 0;
-		if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY_V1, &geo) < 0) {
-			fprintf(stderr, _(
-				"%s: cannot determine geometry of filesystem"
-				" mounted at %s: %s\n"),
-				progname, fname, strerror(errno));
-			exit(1);
+	memset(&geo, '\0', sizeof(geo));
+	geo.version = XFS_FSOP_GEOM_VERSION5;
+	if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY, &geo) < 0) {
+	
+		oldioctl = 1;
+		/* get the current filesystem size & geometry */
+		if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY_V2, &geo) < 0) {
+			/*
+			 * OK, new xfsctl barfed - back off and try
+			 * earlier version as we're probably running an
+			 * older kernel version.  Only field added in
+			 * the v2 geometry xfsctl is "logsunit" so we'll
+			 * zero that out for later display (as zero).
+			 */
+			geo.logsunit = 0;
+			if (xfsctl(fname, ffd, XFS_IOC_FSGEOMETRY_V1, &geo)
+					< 0) {
+				fprintf(stderr,
+	_("%s: cannot determine geometry of filesystem mounted at %s: %s\n"),
+					progname, fname, strerror(errno));
+				exit(1);
+			}
 		}
 	}
 	isint = geo.logstart > 0;
@@ -247,11 +283,12 @@ main(int argc, char **argv)
 	crcs_enabled = geo.flags & XFS_FSOP_GEOM_FLAGS_V5SB ? 1 : 0;
 	ftype_enabled = geo.flags & XFS_FSOP_GEOM_FLAGS_FTYPE ? 1 : 0;
 	finobt_enabled = geo.flags & XFS_FSOP_GEOM_FLAGS_FINOBT ? 1 : 0;
+	utf8 = geo.flags & XFS_FSOP_GEOM_FLAGS_UTF8 ? 1 : 0;
 	if (nflag) {
-		report_info(geo, datadev, isint, logdev, rtdev,
+		report_info(geo, oldioctl, datadev, isint, logdev, rtdev,
 				lazycount, dirversion, logversion,
 				attrversion, projid32bit, crcs_enabled, ci,
-				ftype_enabled, finobt_enabled);
+				ftype_enabled, finobt_enabled, utf8);
 		exit(0);
 	}
 
@@ -286,10 +323,10 @@ main(int argc, char **argv)
 		exit(1);
 	}
 
-	report_info(geo, datadev, isint, logdev, rtdev,
+	report_info(geo, oldioctl, datadev, isint, logdev, rtdev,
 			lazycount, dirversion, logversion,
 			attrversion, projid32bit, crcs_enabled, ci, ftype_enabled,
-			finobt_enabled);
+			finobt_enabled, utf8);
 
 	ddsize = xi.dsize;
 	dlsize = ( xi.logBBsize? xi.logBBsize :
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#49 lore raw

PATCH 32/35 xfsprogs: add utf8 support to mkfs.xfs

In reply to Ben Myers

From: Mark Tinguely <[email protected]>

Set the utf-8 feature bit.

[v2: add support for utf8version. -bpm]

---
 man/man8/mkfs.xfs.8 |  9 ++++-
 mkfs/xfs_mkfs.c     | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 mkfs/xfs_mkfs.h     |  3 +-
 3 files changed, 105 insertions(+), 5 deletions(-)
man/man8/mkfs.xfs.8 +8 -1
index ad9ff3d..aa43cf5 100644
@@ -558,7 +558,7 @@ any power of 2 size from the filesystem block size up to 65536.
 .IP
 The
 .B version=ci
-option enables ASCII only case-insensitive filename lookup and version
+option enables ASCII or UTF-8 case-insensitive filename lookup and version
 2 directories. Filenames are case-preserving, that is, the names
 are stored in directories using the case they were created with.
 .IP
@@ -582,6 +582,13 @@ When CRCs are enabled via
 the ftype functionality is always enabled. This feature can not be turned
 off for such filesystem configurations.
 .IP
+.TP
+.BI utf8[= value ]
+This is used to enable the UTF-8 character set support. The
+.I value
+is either 0 or 1, with 1 signifying that UTF-8 character support is to be
+enabled. If the value is omitted, 1 is assumed.
+.IP
 .RE
 .TP
 .BI \-p " protofile"
mkfs/xfs_mkfs.c +95 -3

176 lines not on this page · raw message

mkfs/xfs_mkfs.h +2 -1
index 9df5f37..f40b284 100644
@@ -37,13 +37,14 @@
 	0 ) : XFS_SB_VERSION_1 )
 
 #define XFS_SB_VERSION2_MKFS(crc, lazycount, attr2, projid32bit, parent, \
-			     ftype) (\
+			     ftype, utf8) (\
 	((lazycount) ? XFS_SB_VERSION2_LAZYSBCOUNTBIT : 0) |		\
 	((attr2) ? XFS_SB_VERSION2_ATTR2BIT : 0) |			\
 	((projid32bit) ? XFS_SB_VERSION2_PROJID32BIT : 0) |		\
 	((parent) ? XFS_SB_VERSION2_PARENTBIT : 0) |			\
 	((crc) ? XFS_SB_VERSION2_CRCBIT : 0) |				\
 	((ftype) ? XFS_SB_VERSION2_FTYPE : 0) |				\
+	((utf8) ? XFS_SB_VERSION2_UTF8BIT : 0) |			\
 	0 )
 
 #define	XFS_DFL_BLOCKSIZE_LOG	12		/* 4096 byte blocks */
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#50 lore raw

PATCH 33/35 xfsprogs: add utf8 support to xfs_repair

In reply to Ben Myers

From: Mark Tinguely <[email protected]>

Fix the duplicate filename detection to use the utf-8 normalization routines.

[XXX use sb_utf8version on the global xfs_mount.
TODO maybe add the xfs_mount to the args structure? --bpm]

---
 db/check.c        |  2 +-
 libxfs/xfs_utf8.c | 16 ++++++++--------
 repair/phase6.c   | 36 +++++++++++++++++++++++++-----------
 3 files changed, 34 insertions(+), 20 deletions(-)
db/check.c +1 -1
index d317a71..9219cc8 100644
@@ -2324,7 +2324,7 @@ process_data_dir_v2(
 			(char *)dep - (char *)data);
 		dir_hash_add(mp->m_dirnameops->hashname(dep->name,
 					dep->namelen,
-					0 /* version for later */), addr);
+					mp->m_sb.sb_utf8version), addr);
 		ptr += xfs_dir3_data_entsize(mp, dep->namelen);
 		count++;
 		lastfree = 0;
libxfs/xfs_utf8.c +8 -8
index f7042ef..e7a717e 100644
@@ -106,8 +106,8 @@ xfs_utf8_normhash(
 	unsigned char	*norm;
 	ssize_t		normlen;
 	int		c;
-	unsigned int	sb_utf8version =
-		args->dp->i_mount->m_sb.sb_utf8version;
+	unsigned int	sb_utf8version = mp->m_sb.sb_utf8version;
+/* XXX		args->dp->i_mount->m_sb.sb_utf8version; */
 
 	/* Don't normalize system attribute names. */
 	if (args->flags & (ATTR_ROOT|ATTR_SECURE))
@@ -147,8 +147,8 @@ xfs_utf8_compname(
 	struct utf8cursor u8c;
 	const unsigned char *norm;
 	int		c;
-	unsigned int	sb_utf8version =
-		args->dp->i_mount->m_sb.sb_utf8version;
+	unsigned int	sb_utf8version = mp->m_sb.sb_utf8version;
+/* XXX		args->dp->i_mount->m_sb.sb_utf8version; */
 
 	ASSERT(args->norm || args->normlen == -1);
 
@@ -213,8 +213,8 @@ xfs_utf8_ci_normhash(
 	unsigned char	*norm;
 	ssize_t		normlen;
 	int		c;
-	unsigned int	sb_utf8version =
-		args->dp->i_mount->m_sb.sb_utf8version;
+	unsigned int	sb_utf8version = mp->m_sb.sb_utf8version;
+/* XXX		args->dp->i_mount->m_sb.sb_utf8version; */
 
 	/* Don't normalize system attribute names. */
 	if (args->flags & (ATTR_ROOT|ATTR_SECURE))
@@ -254,8 +254,8 @@ xfs_utf8_ci_compname(
 	struct utf8cursor u8c;
 	const unsigned char *norm;
 	int		c;
-	unsigned int	sb_utf8version =
-		args->dp->i_mount->m_sb.sb_utf8version;
+	unsigned int	sb_utf8version = mp->m_sb.sb_utf8version;
+/* XXX		args->dp->i_mount->m_sb.sb_utf8version; */
 
 	ASSERT(args->norm || args->normlen == -1);
 
repair/phase6.c +25 -11
index c18ef69..eb3ea35 100644
@@ -176,13 +176,15 @@ dir_hash_add(
 	unsigned char		*name,
 	__uint8_t		ftype)
 {
-	xfs_dahash_t		hash = 0;
 	int			byaddr;
 	int			byhash = 0;
 	dir_hash_ent_t		*p;
 	int			dup;
 	short			junk;
 	struct xfs_name		xname;
+	xfs_da_args_t		args;
+
+	memset(&args, 0, sizeof(xfs_da_args_t));
 
 	ASSERT(!hashtab->names_duped);
 
@@ -195,20 +197,30 @@ dir_hash_add(
 	dup = 0;
 
 	if (!junk) {
-		hash = mp->m_dirnameops->hashname(name, namelen,
-				0 /* version for later */);
-		byhash = DIR_HASH_FUNC(hashtab, hash);
+		int error;
+
+		args.name = name;
+		args.namelen = namelen;
+		args.inumber = inum;
+		args.whichfork = XFS_DATA_FORK;
+
+		error = mp->m_dirnameops->normhash(&args);
+		if (error)
+			do_error(_("normalize has failed %d)\n"), error);
+
+		byhash = DIR_HASH_FUNC(hashtab, args.hashval);
 
 		/*
 		 * search hash bucket for existing name.
 		 */
 		for (p = hashtab->byhash[byhash]; p; p = p->nextbyhash) {
-			if (p->hashval == hash && p->name.len == namelen) {
-				if (memcmp(p->name.name, name, namelen) == 0) {
-					dup = 1;
-					junk = 1;
-					break;
-				}
+			if (p->hashval == args.hashval &&
+			    mp->m_dirnameops->compname(&args, p->name.name,
+						       p->name.len) !=
+							 XFS_CMP_DIFFERENT) {
+				dup = 1;
+				junk = 1;
+				break;
 			}
 		}
 	}
@@ -227,7 +239,7 @@ dir_hash_add(
 	hashtab->last = p;
 
 	if (!(p->junkit = junk)) {
-		p->hashval = hash;
+		p->hashval = args.hashval;
 		p->nextbyhash = hashtab->byhash[byhash];
 		hashtab->byhash[byhash] = p;
 	}
@@ -236,6 +248,8 @@ dir_hash_add(
 	p->seen = 0;
 	p->name = xname;
 
+	if (args.norm)
+		kmem_free((void *) args.norm);
 	return !dup;
 }
 
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#51 lore raw

PATCH 34/35 xfsprogs: xfs_db support for sb_utf8version

In reply to Ben Myers

From: Ben Myers <[email protected]>

Add support for accessing and setting sb_utf8version to xfs_db.

---
 db/hash.c | 1 +
 db/sb.c   | 3 +++
 2 files changed, 4 insertions(+)
db/hash.c +1 -0
index 02376e6..5196442 100644
@@ -52,6 +52,7 @@ hash_f(
 {
 	xfs_dahash_t	hashval;
 
+	/* XXX utf8version? */
 	hashval = libxfs_da_hashname((uchar_t *)argv[1], (int)strlen(argv[1]));
 	dbprintf("0x%x\n", hashval);
 	return 0;
db/sb.c +3 -0
index 6cb665d..e32790a 100644
@@ -119,6 +119,7 @@ const field_t	sb_flds[] = {
 	{ "features_log_incompat", FLDT_UINT32X, OI(OFF(features_log_incompat)),
 		C1, 0, TYP_NONE },
 	{ "crc", FLDT_CRC, OI(OFF(crc)), C1, 0, TYP_NONE },
+	{ "utf8version", FLDT_UINT32D, OI(OFF(utf8version)), C1, 0, TYP_NONE },
 	{ "pquotino", FLDT_INO, OI(OFF(pquotino)), C1, 0, TYP_INODE },
 	{ "lsn", FLDT_UINT64X, OI(OFF(lsn)), C1, 0, TYP_NONE },
 	{ NULL }
@@ -646,6 +647,8 @@ version_string(
 		strcat(s, ",CRC");
 	if (xfs_sb_version_hasftype(sbp))
 		strcat(s, ",FTYPE");
+	if (xfs_sb_version_hasutf8(sbp))
+		strcat(s, ",UTF8");
 	return s;
 }
 
Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#52 lore raw

PATCH 35/35 xfsprogs: add a test for utf8 support

In reply to Ben Myers

From: Ben Myers <[email protected]>

Here's a basic test for utf8 support in xfs. It is based on code that does testing in the trie generator. Here too we are using the NormalizationTest-7.0.0.txt file from the unicode distribution. We check that the normalization in libxfs is working and then run checks on a filesystem mounted on /mnt (currently this is hardcoded). Note that there are some 'blacklisted' unichars which normalize to reserved characters.

---
 Makefile                  |   2 +-
 chkutf8data/Makefile      |  21 +++
 chkutf8data/chkutf8data.c | 451 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 473 insertions(+), 1 deletion(-)
 create mode 100644 chkutf8data/Makefile
 create mode 100644 chkutf8data/chkutf8data.c
Makefile +1 -1
index 74778b5..d2be322 100644
@@ -42,7 +42,7 @@ endif
 
 LIB_SUBDIRS = utf8norm libxfs libxlog libxcmd libhandle libdisk
 TOOL_SUBDIRS = copy db estimate fsck fsr growfs io logprint mkfs quota \
-		mdrestore repair rtcp m4 man doc po debian
+		mdrestore repair rtcp m4 man doc po debian chkutf8data
 
 SUBDIRS = include $(LIB_SUBDIRS) $(TOOL_SUBDIRS)
 
chkutf8data/Makefile new file +21 -0
new file mode 100644
index 0000000..6ce5706
@@ -0,0 +1,21 @@
+#
+# Copyright (c) 2014 SGI. All Rights Reserved.
+#
+
+TOPDIR = ..
+include $(TOPDIR)/include/builddefs
+
+LTCOMMAND = chkutf8data
+CFILES = chkutf8data.c
+
+LLDLIBS = $(LIBXFS)
+LTDEPENDENCIES = $(LIBXFS)
+LLDFLAGS = -static
+
+default: depend $(LTCOMMAND)
+
+include $(BUILDRULES)
+
+install: default
+
+-include .ltdep
chkutf8data/chkutf8data.c new file +451 -0

451 lines not on this page · raw message

Signature · 6 lines
1.7.12.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#25 lore raw

Re: [PATCH 13/16] xfs: implement demand load of utf8norm.ko

In reply to Ben Myers
Quoted 8 lines
fs/xfs/libxfs/xfs_utf8.c +8 -0
+int
+xfs_init_utf8_module(struct xfs_mount	*mp)
+{
+	request_module("utf8norm");
+
+	spin_lock(&utf8norm_lock);
+	if (utf8norm_initialized) {
+		spin_unlock(&utf8norm_lock);

fs/xfs/libxfs/xfs_utf8.c line 110

Quoted 10 lines
fs/xfs/libxfs/xfs_utf8.c +10 -0
+		return 0;
+	}
+
+	utf8version_is_supported_func = symbol_get(utf8version_is_supported);
+	if (!utf8version_is_supported_func)
+		goto error;
+
+	utf8nfkdi_func = symbol_get(utf8nfkdi);
+	if (!utf8nfkdi_func)
+		goto error;

Please export a structure with a function pointes so that we just need a single symbol_get call. I'd have to look up how symbol_get works, but unless there's something that speaks against this it might be simpler to than just do a symbol_get per mounst structure that uses utf8 and can point to that structure so that we don't have to add additional reference counting infrastructure around it.

#5 lore raw

Re: [PATCH 04/16] lib/utf8norm.c: reduce the size of utf8data[]

In reply to Ben Myers
Quoted 10 lines · Ben Myers

From: Olaf Weber <[email protected]>

Remove the Hangul decompositions from the utf8data trie, and do algorithmic decomposition to calculate them on the fly. To store the decomposition the caller of utf8lookup()/utf8nlookup() must provide a 12-byte buffer, which is used to synthesize a leaf with the decomposition. Trie size is reduced from 245kB to 90kB.

This change also contains a number of robustness fixes to the trie generator mkutf8data.c.

Please separate out the robustness fixes or merge them back into the original patch. e.g. Bulk renaming of code like this:

Quoted 12 lines

static int

scripts/mkutf8data.c +0 -11
-utf8key(unsigned int key, char keyval[])
-{
-	int keylen;
-
-	if (key < 0x80) {
-		keyval[0] = key;
-		keylen = 1;
-	} else if (key < 0x800) {
-		keyval[1] = key & UTF8_V_MASK;
-		keyval[1] |= UTF8_N_BITS;
-		key >>= UTF8_V_SHIFT;

....

scripts/mkutf8data.c utf8trie_t *nfkdicf; · line 271

Quoted 11 lines
scripts/mkutf8data.c +11 -0
+utf8encode(char *str, unsigned int val)
+{
+	int len;
+
+	if (val < 0x80) {
+		str[0] = val;
+		len = 1;
+	} else if (val < 0x800) {
+		str[1] = val & UTF8_V_MASK;
+		str[1] |= UTF8_N_BITS;
+		val >>= UTF8_V_SHIFT;

Doesn't belong in a patch that introduces special hangul character handling....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#28 lore raw

Re: [PATCH 14/16] xfs: rename XFS_IOC_FSGEOM to XFS_IOC_FSGEOM_V2

In reply to Ben Myers
Quoted 4 lines · Ben Myers

From: Ben Myers <[email protected]>

We'll be creating a new versioned XFS_IOC_FSGEOMETRY ioctl and structure so rename the current revision to _V2.

Urk, no.

This will result in older applications picking up the new ioctl when they are rebuilt without having explicit support for the new ioctl.

Just create a new ioctl with a new name and modify applications to use it. If the kernel does not support the new ioctl, then fall back to the old one in userspace.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#29 lore raw

Re: [PATCH 14/16] xfs: rename XFS_IOC_FSGEOM to XFS_IOC_FSGEOM_V2

Quoted 6 lines · Dave Chinner
Quoted 4 lines · Ben Myers

From: Ben Myers <[email protected]>

We'll be creating a new versioned XFS_IOC_FSGEOMETRY ioctl and structure so rename the current revision to _V2.

Quoted 8 lines

Urk, no.

This will result in older applications picking up the new ioctl when they are rebuilt without having explicit support for the new ioctl.

Just create a new ioctl with a new name and modify applications to use it. If the kernel does not support the new ioctl, then fall back to the old one in userspace.

D'oh. Yeah, ok.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#31 lore raw

Re: [PATCH 15/16] xfs: xfs_fs_geometry returns a number of bytes to copy

In reply to Ben Myers
Quoted 5 lines · Ben Myers

From: Ben Myers <[email protected]>

The versioned xfs_fsop_geom_t will be of variable size. Make xfs_fs_geometry return the number of bytes to copy out to userspace for a given version of the structure.

xfs_fs_geometry() should be a void right now - it doesn't return any error value at all.

Further, the size of the structure that is filled in is determined by the version of the ioctl being called, not the xfs_fs_geometry() function. Hence the caller already knows the size of the structure being used, and hence does not need xfs_fs_geometry() to tell it that information.

So I don't think this change is necessary.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#33 lore raw

Re: [PATCH 16/16] xfs: add versioned fsgeom ioctl with utf8version field

In reply to Ben Myers
Quoted 40 lines · Ben Myers

From: Ben Myers <[email protected]>

This adds a utf8version field to the xfs_fs_geom structure. An important characteristic of this version of the ioctl is that fsgeo.version needs to be set by the caller to specify which version of the structure to return.

---
 fs/xfs/xfs_fs.h    | 31 +++++++++++++++++++++++++++++++
 fs/xfs/xfs_fsops.c | 13 ++++++++++++-
 fs/xfs/xfs_fsops.h |  2 +-
 fs/xfs/xfs_ioctl.c | 31 +++++++++++++++++++++++++++++++
 4 files changed, 75 insertions(+), 2 deletions(-)
fs/xfs/xfs_fs.h +17 -0
index fd45cbe..2f4d430 100644
@@ -206,6 +206,34 @@ typedef struct xfs_fsop_geom_v2 {
 	__u32		logsunit;	/* log stripe unit, bytes */
 } xfs_fsop_geom_v2_t;
 
+/*
+ * Output for XFS_IOC_FSGEOMETRY
+ */
+typedef struct xfs_fsop_geom {
+	__u32		blocksize;	/* filesystem (data) block size */
+	__u32		rtextsize;	/* realtime extent size		*/
+	__u32		agblocks;	/* fsblocks in an AG		*/
+	__u32		agcount;	/* number of allocation groups	*/
+	__u32		logblocks;	/* fsblocks in the log		*/
+	__u32		sectsize;	/* (data) sector size, bytes	*/
+	__u32		inodesize;	/* inode size in bytes		*/
+	__u32		imaxpct;	/* max allowed inode space(%)	*/
+	__u64		datablocks;	/* fsblocks in data subvolume	*/
+	__u64		rtblocks;	/* fsblocks in realtime subvol	*/
+	__u64		rtextents;	/* rt extents in realtime subvol*/
+	__u64		logstart;	/* starting fsblock of the log	*/
+	unsigned char	uuid[16];	/* unique id of the filesystem	*/

fs/xfs/xfs_fs.h typedef struct xfs_fsop_geom_v2 { · line 235

Quoted 10 lines
fs/xfs/xfs_fs.h +10 -0
+	__u32		sunit;		/* stripe unit, fsblocks	*/
+	__u32		swidth;		/* stripe width, fsblocks	*/
+	__s32		version;	/* structure version		*/
+	__u32		flags;		/* superblock version flags	*/
+	__u32		logsectsize;	/* log sector size, bytes	*/
+	__u32		rtsectsize;	/* realtime sector size, bytes	*/
+	__u32		dirblocksize;	/* directory block size, bytes	*/
+	__u32		logsunit;	/* log stripe unit, bytes */
+	__u32		utf8version;	/* Unicode version		*/
+} xfs_fsop_geom_t;

New structure defintion, not a redefinition of the old name, please. Drop the typedef, and the structure needs to be 64 bit size aligned so we don't get problems with 32 bit userspace on 64 bit kernels (e.g. we have a v1 compat ioctl handler because of this issue).

Further, lets avoid needing to rev the ioctl again in future by adding a bunch of "must be zero" padding to the new structure so we can extend the information we push to userspace easily. i.e. padding only becomes meaningful when the superblock flag that exposes meaning is set. i.e. userspace can do this to conditionally access the ut8version value if it is meaningful:

	utf8_ver = 0;
	if (geo.flags & XFS_FSOP_GEOM_FLAGS_UTF8)
		utf8_ver = geo->utf8version;

i.e. let's make the new structure forwards compatible with new features...

fs/xfs/xfs_fs.h xfs_fs_geometry( · line 127

Quoted 12 lines
fs/xfs/xfs_fs.h +8 -0
@@ -115,6 +117,15 @@ xfs_fs_geometry(
 				XFS_FSOP_GEOM_FLAGS_LOGV2 : 0);
 		geo->logsunit = mp->m_sb.sb_logsunit;
 	}
+	if (new_version >= XFS_FSOP_GEOM_VERSION5) {
+		geo->version = XFS_FSOP_GEOM_VERSION5;
+		geo->flags |= (xfs_sb_version_hasutf8(&mp->m_sb) ?
+				XFS_FSOP_GEOM_FLAGS_UTF8 : 0);
+		geo->utf8version = mp->m_sb.sb_utf8version;
+		if (bytes)
+			*bytes = sizeof(xfs_fsop_geom_v2_t) +
+				 sizeof(geo->utf8version);

Further indication that the *bytes variable should die.

Quoted 27 lines
fs/xfs/xfs_fs.h +1 -0
+	}
 	return 0;
 }
 
fs/xfs/xfs_fsops.h +1 -1
index 74e1fee..b723f36 100644
@@ -18,7 +18,7 @@
 #ifndef __XFS_FSOPS_H__
 #define	__XFS_FSOPS_H__
 
-extern int xfs_fs_geometry(xfs_mount_t *mp, xfs_fsop_geom_v2_t *geo,
+extern int xfs_fs_geometry(xfs_mount_t *mp, void *buffer,
 		int new_version, size_t *bytes);
 extern int xfs_growfs_data(xfs_mount_t *mp, xfs_growfs_data_t *in);
 extern int xfs_growfs_log(xfs_mount_t *mp, xfs_growfs_log_t *in);
fs/xfs/xfs_ioctl.c +2 -0
index 1657ce5..6308680 100644
@@ -859,6 +859,34 @@ xfs_ioc_fsgeometry_v2(
 	return 0;
 }
 
+STATIC int
+xfs_ioc_fsgeometry(

fs/xfs/xfs_ioctl.c xfs_ioc_fsgeometry_v2( · line 873

Quoted 10 lines
fs/xfs/xfs_ioctl.c +10 -0
+	struct xfs_mount	*mp,
+	void			__user *arg)
+{
+	xfs_fsop_geom_t		fsgeo;
+	int			version, error;
+	size_t			bytes;
+
+	/* offsetof(version)? XXX just get 32 bits? */
+	if (copy_from_user(&fsgeo, arg, sizeof(xfs_fsop_geom_v1_t)))
+		return -EFAULT;

It's best to copy in the entire structure rather than play offset games.

fs/xfs/xfs_ioctl.c xfs_ioc_fsgeometry_v2( · line 878

Quoted 4 lines
fs/xfs/xfs_ioctl.c +4 -0
+	version = fsgeo.version;
+
+	if (version < XFS_FSOP_GEOM_VERSION5)
+		return -EINVAL;

Here it rejects anything that is not a v3 structure aware of the unicode extensions, which means it breaks any old recompiled application that hasn't been updated to support XFS_FSOP_GEOM_VERSION5 despite the fact that they will compile against headers with the new definition without warnings or errors.

fs/xfs/xfs_ioctl.c xfs_ioc_fsgeometry_v2( · line 886

Quoted 8 lines
fs/xfs/xfs_ioctl.c +8 -0
+
+	memset(&fsgeo, 0, sizeof(fsgeo));
+	error = xfs_fs_geometry(mp, &fsgeo, version, &bytes);
+	if (error)
+		return error;
+
+	if (copy_to_user(arg, &fsgeo, bytes))
+		return -EFAULT;

and you can use sizeof(struct xfs_fs_geom_v3) here instead of bytes.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#17 lore raw

Re: [PATCH 09/16] xfs: add a superblock feature bit to indicate UTF-8 support.

In reply to Ben Myers
Quoted 20 lines · Ben Myers

From: Olaf Weber <[email protected]>

When UTF-8 support is enabled, the xfs_dir_ci_inode_operations must be installed. Add xfs_sb_version_hasci(), which tests both the borgbit and the utf8bit, and returns true if at least one of them is set. Replace calls to xfs_sb_version_hasasciici() as needed.

---
 fs/xfs/libxfs/xfs_sb.h | 24 +++++++++++++++++++++++-
 fs/xfs/xfs_fs.h        |  1 +
 fs/xfs/xfs_fsops.c     |  4 +++-
 fs/xfs/xfs_iops.c      |  4 ++--
 4 files changed, 29 insertions(+), 4 deletions(-)
fs/xfs/libxfs/xfs_sb.h +0 -0
index 2e73970..525eacb 100644

fs/xfs/libxfs/xfs_sb.h struct xfs_trans; · line 74

Quoted 5 lines
fs/xfs/libxfs/xfs_sb.h +1 -0
@@ -70,6 +70,7 @@ struct xfs_trans;
 #define XFS_SB_VERSION2_RESERVED4BIT	0x00000004
 #define XFS_SB_VERSION2_ATTR2BIT	0x00000008	/* Inline attr rework */
 #define XFS_SB_VERSION2_PARENTBIT	0x00000010	/* parent pointers */
+#define XFS_SB_VERSION2_UTF8BIT		0x00000020      /* utf8 names */
 #define XFS_SB_VERSION2_PROJID32BIT	0x00000080	/* 32 bit project id */

Can you explain why this bit is safe to use? I don't recall why XFS_SB_VERSION2_PROJID32BIT skipped several bits because there aren't any comments explaining why that value was chosen. Adding a comment about the 0x00000040 bit at the same time would be useful.

Quoted 11 lines
fs/xfs/libxfs/xfs_sb.h +1 -0
 #define XFS_SB_VERSION2_CRCBIT		0x00000100	/* metadata CRCs */
 #define XFS_SB_VERSION2_FTYPE		0x00000200	/* inode type in dir */
@@ -77,6 +78,7 @@ struct xfs_trans;
 #define	XFS_SB_VERSION2_OKBITS		\
 	(XFS_SB_VERSION2_LAZYSBCOUNTBIT	| \
 	 XFS_SB_VERSION2_ATTR2BIT	| \
+	 XFS_SB_VERSION2_UTF8BIT	| \
 	 XFS_SB_VERSION2_PROJID32BIT	| \
 	 XFS_SB_VERSION2_FTYPE)
 

fs/xfs/libxfs/xfs_sb.h xfs_sb_has_ro_compat_feature( · line 517

Quoted 8 lines
fs/xfs/libxfs/xfs_sb.h +3 -1
@@ -509,8 +511,10 @@ xfs_sb_has_ro_compat_feature(
 }
 
 #define XFS_SB_FEAT_INCOMPAT_FTYPE	(1 << 0)	/* filetype in dirent */
+#define XFS_SB_FEAT_INCOMPAT_UTF8	(1 << 1)	/* utf-8 name support */
 #define XFS_SB_FEAT_INCOMPAT_ALL \
-		(XFS_SB_FEAT_INCOMPAT_FTYPE)
+		(XFS_SB_FEAT_INCOMPAT_FTYPE | \
+		 XFS_SB_FEAT_INCOMPAT_UTF8)

Don't add support to the filesystem until all the supporting code is in place. This avoids git bisects landing on commits in the series where the filesystem says it supports the feature bit it doesn't actually work. Add a patch at the end of the series that adds these bits to the feature masks.

fs/xfs/libxfs/xfs_sb.h static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp) · line 565

Quoted 8 lines
fs/xfs/libxfs/xfs_sb.h +1 -0
 #define XFS_SB_FEAT_INCOMPAT_UNKNOWN	~XFS_SB_FEAT_INCOMPAT_ALL
 static inline bool
@@ -558,6 +562,24 @@ static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp)
 		(sbp->sb_features_ro_compat & XFS_SB_FEAT_RO_COMPAT_FINOBT);
 }
 
+static inline int xfs_sb_version_hasutf8(xfs_sb_t *sbp)

bool, no typedefs.

fs/xfs/libxfs/xfs_sb.h static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp) · line 570

Quoted 5 lines
fs/xfs/libxfs/xfs_sb.h +5 -0
+{
+	return (XFS_SB_VERSION_NUM(sbp) == XFS_SB_VERSION_5 &&
+		xfs_sb_has_incompat_feature(sbp, XFS_SB_FEAT_INCOMPAT_UTF8)) ||
+		(xfs_sb_version_hasmorebits(sbp) &&
+		(sbp->sb_features2 & XFS_SB_VERSION2_UTF8BIT));

xfs_sb_version_hasmorebits() already checks for XFS_SB_VERSION_5, so this could be:

	return xfs_sb_version_hasmorebits(sbp) &&
		(xfs_sb_has_incompat_feature(sbp, XFS_SB_FEAT_INCOMPAT_UTF8) ||
		 (sbp->sb_features2 & XFS_SB_VERSION2_UTF8BIT));

fs/xfs/libxfs/xfs_sb.h static inline int xfs_sb_version_hasfinobt(xfs_sb_t *sbp) · line 579

Quoted 9 lines
fs/xfs/libxfs/xfs_sb.h +9 -0
+}
+
+/*
+ * Special case: there are a number of places where we need to test
+ * both the borgbit and the utf8bit, and take the same action if
+ * either of those is set.
+ */
+static inline int xfs_sb_version_hasci(xfs_sb_t *sbp)
+{

bool, no typedefs, and probably should be a separate patch.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#20 lore raw

Re: [PATCH 10/16] xfs: store utf8version in the superblock

In reply to Ben Myers
Quoted 40 lines · Ben Myers

From: Ben Myers <[email protected]>

The utf8 version a filesystem was created with needs to be stored in order that normalizations will remain stable over the lifetime of the filesystem. Convert sb_pad to sb_utf8version in the super block. This also adds checks at mount time to see whether the unicode normalization module has support for the version of unicode that the filesystem requires. If not we fail the mount.

---
 fs/xfs/libxfs/xfs_dir2.c | 28 ++++++++++++++++---
 fs/xfs/libxfs/xfs_sb.c   |  4 +--
 fs/xfs/libxfs/xfs_sb.h   | 10 ++++---
 fs/xfs/libxfs/xfs_utf8.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_utf8.h | 24 +++++++++++++++++
 5 files changed, 126 insertions(+), 10 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_utf8.c
 create mode 100644 fs/xfs/libxfs/xfs_utf8.h
fs/xfs/libxfs/xfs_dir2.c +8 -4
index 4eb0973..2c89211 100644
@@ -157,10 +157,30 @@ xfs_da_mount(
 				(uint)sizeof(xfs_da_node_entry_t);
 	dageo->magicpct = (dageo->blksize * 37) / 100;
 
-	if (xfs_sb_version_hasasciici(&mp->m_sb))
-		mp->m_dirnameops = &xfs_ascii_ci_nameops;
-	else
-		mp->m_dirnameops = &xfs_default_nameops;
+	if (xfs_sb_version_hasutf8(&mp->m_sb)) {
+#ifdef CONFIG_XFS_UTF8
+		if (!xfs_utf8_version_ok(mp))
+			return -ENOSYS;
+
+		/* XXX these are replaced in the next patch need
+		   to do some kind of reordering here */
+		if (xfs_sb_version_hasasciici(&mp->m_sb))

fs/xfs/libxfs/xfs_dir2.c xfs_da_mount( · line 177

Quoted 10 lines
fs/xfs/libxfs/xfs_dir2.c +10 -0
+			mp->m_dirnameops = &xfs_ascii_ci_nameops;
+		else
+			mp->m_dirnameops = &xfs_default_nameops;
+#else
+		xfs_warn(mp,
+"Recompile XFS with CONFIG_XFS_UTF8 to mount this filesystem");
+		kmem_free(mp->m_dir_geo);
+		kmem_free(mp->m_attr_geo);
+		return -ENOSYS;
+#endif

This config check doesn't belong here. Validation of superblock fields vs the current config goes in the superblock verifier. I also think that indication of UTF8 support being compiled in needs to go in the XFS_VERSION_STRING, not have ifdef hackery added into the code.

i.e. the mount should fail very early on with a superblock verification failure from xfs_mount_validate_sb().

Quoted 71 lines
fs/xfs/libxfs/xfs_dir2.c +6 -0
+	} else {
+		if (xfs_sb_version_hasasciici(&mp->m_sb))
+			mp->m_dirnameops = &xfs_ascii_ci_nameops;
+		else
+			mp->m_dirnameops = &xfs_default_nameops;
+	}
 
 	return 0;
 }
fs/xfs/libxfs/xfs_sb.c +2 -2
index ad525a5..1ee7d33 100644
@@ -99,7 +99,7 @@ static const struct {
 	{ offsetof(xfs_sb_t, sb_features_incompat),	0 },
 	{ offsetof(xfs_sb_t, sb_features_log_incompat),	0 },
 	{ offsetof(xfs_sb_t, sb_crc),		0 },
-	{ offsetof(xfs_sb_t, sb_pad),		0 },
+	{ offsetof(xfs_sb_t, sb_utf8version),	0 },
 	{ offsetof(xfs_sb_t, sb_pquotino),	0 },
 	{ offsetof(xfs_sb_t, sb_lsn),		0 },
 	{ sizeof(xfs_sb_t),			0 }
@@ -443,7 +443,7 @@ __xfs_sb_from_disk(
 	to->sb_features_incompat = be32_to_cpu(from->sb_features_incompat);
 	to->sb_features_log_incompat =
 				be32_to_cpu(from->sb_features_log_incompat);
-	to->sb_pad = 0;
+	to->sb_utf8version = be32_to_cpu(from->sb_utf8version);
 	to->sb_pquotino = be64_to_cpu(from->sb_pquotino);
 	to->sb_lsn = be64_to_cpu(from->sb_lsn);
 	/* Convert on-disk flags to in-memory flags? */
fs/xfs/libxfs/xfs_sb.h +4 -3
index 525eacb..dc7b6c6 100644
@@ -159,7 +159,7 @@ typedef struct xfs_sb {
 	__uint32_t	sb_features_log_incompat;
 
 	__uint32_t	sb_crc;		/* superblock crc */
-	__uint32_t	sb_pad;
+	__uint32_t	sb_utf8version;	/* unicode version */
 
 	xfs_ino_t	sb_pquotino;	/* project quota inode */
 	xfs_lsn_t	sb_lsn;		/* last write sequence */
@@ -245,7 +245,7 @@ typedef struct xfs_dsb {
 	__be32		sb_features_log_incompat;
 
 	__le32		sb_crc;		/* superblock crc */
-	__be32		sb_pad;
+	__be32		sb_utf8version;	/* version of unicode */
 
 	__be64		sb_pquotino;	/* project quota inode */
 	__be64		sb_lsn;		/* last write sequence */
@@ -271,7 +271,7 @@ typedef enum {
 	XFS_SBS_LOGSECTLOG, XFS_SBS_LOGSECTSIZE, XFS_SBS_LOGSUNIT,
 	XFS_SBS_FEATURES2, XFS_SBS_BAD_FEATURES2, XFS_SBS_FEATURES_COMPAT,
 	XFS_SBS_FEATURES_RO_COMPAT, XFS_SBS_FEATURES_INCOMPAT,
-	XFS_SBS_FEATURES_LOG_INCOMPAT, XFS_SBS_CRC, XFS_SBS_PAD,
+	XFS_SBS_FEATURES_LOG_INCOMPAT, XFS_SBS_CRC, XFS_SBS_UTF8VERSION,
 	XFS_SBS_PQUOTINO, XFS_SBS_LSN,
 	XFS_SBS_FIELDCOUNT
 } xfs_sb_field_t;
@@ -303,6 +303,7 @@ typedef enum {
 #define XFS_SB_FEATURES_INCOMPAT XFS_SB_MVAL(FEATURES_INCOMPAT)
 #define XFS_SB_FEATURES_LOG_INCOMPAT XFS_SB_MVAL(FEATURES_LOG_INCOMPAT)
 #define XFS_SB_CRC		XFS_SB_MVAL(CRC)
+#define XFS_SB_UTF8VERSION	XFS_SB_MVAL(UTF8VERSION)
 #define XFS_SB_PQUOTINO		XFS_SB_MVAL(PQUOTINO)
 #define	XFS_SB_NUM_BITS		((int)XFS_SBS_FIELDCOUNT)
 #define	XFS_SB_ALL_BITS		((1LL << XFS_SB_NUM_BITS) - 1)

fs/xfs/libxfs/xfs_sb.h typedef enum { · line 318

Quoted 6 lines
fs/xfs/libxfs/xfs_sb.h +2 -1
@@ -313,7 +314,8 @@ typedef enum {
 	 XFS_SB_ICOUNT | XFS_SB_IFREE | XFS_SB_FDBLOCKS | XFS_SB_FEATURES2 | \
 	 XFS_SB_BAD_FEATURES2 | XFS_SB_FEATURES_COMPAT | \
 	 XFS_SB_FEATURES_RO_COMPAT | XFS_SB_FEATURES_INCOMPAT | \
-	 XFS_SB_FEATURES_LOG_INCOMPAT | XFS_SB_PQUOTINO)
+	 XFS_SB_FEATURES_LOG_INCOMPAT | XFS_SB_UTF8VERSION | \
+	 XFS_SB_PQUOTINO)

We should never be modifying the utf8 version number from the kernel, so this shouldn't be set in the XFS_SB_MOD_BITS mask.

Quoted 39 lines
fs/xfs/libxfs/xfs_utf8.c new file +33 -0
new file mode 100644
index 0000000..7e63111
@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2014 SGI.
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#include "xfs.h"
+#include "xfs_fs.h"
+#include "xfs_types.h"
+#include "xfs_bit.h"
+#include "xfs_log_format.h"
+#include "xfs_inum.h"
+#include "xfs_trans.h"
+#include "xfs_trans_resv.h"
+#include "xfs_sb.h"
+#include "xfs_ag.h"
+#include "xfs_da_format.h"
+#include "xfs_da_btree.h"
+#include "xfs_dir2.h"
+#include "xfs_mount.h"
+#include "xfs_da_btree.h"

fs/xfs/libxfs/xfs_utf8.c line 43

Quoted 10 lines
fs/xfs/libxfs/xfs_utf8.c new file +10 -0
+#include "xfs_format.h"
+#include "xfs_bmap_btree.h"
+#include "xfs_alloc_btree.h"
+#include "xfs_dinode.h"
+#include "xfs_inode.h"
+#include "xfs_inode_item.h"
+#include "xfs_bmap.h"
+#include "xfs_error.h"
+#include "xfs_trace.h"
+#include "xfs_utf8.h"

This may sound pedantic, but in all the libxfs rework I've managed to standadise the initial include file order to be roughly:

#include "xfs.h"
#include "xfs_fs.h"
#include "xfs_shared.h"
#include "xfs_format.h"
#include "xfs_log_format.h"
#include "xfs_trans_resv.h"
#include "xfs_bit.h"
#include "xfs_inum.h"
#include "xfs_sb.h"
#include "xfs_ag.h"
#include "xfs_mount.h"
#include "xfs_da_format.h"

i.e. include all the type, shared and on-disk format information first. Can you re-order these to follow the same convention?

fs/xfs/libxfs/xfs_utf8.c line 44

Quoted 1 line
fs/xfs/libxfs/xfs_utf8.c new file +1 -0
+#include <linux/utf8norm.h>

And that should end up being included from fs/xfs/xfs_linux.h, because we can't include things directly from the linux kernel headers in fs/xfs/libxfs/ files.

fs/xfs/libxfs/xfs_utf8.c line 46

Quoted 2 lines
fs/xfs/libxfs/xfs_utf8.c new file +2 -0
+
+int

Bool.

fs/xfs/libxfs/xfs_utf8.c line 53

Quoted 7 lines
fs/xfs/libxfs/xfs_utf8.c new file +7 -0
+xfs_utf8_version_ok(
+	struct xfs_mount	*mp)
+{
+	int	major, minor, revision;
+
+	if (utf8version_is_supported(mp->m_sb.sb_utf8version))
+		return 1;

return true;

Quoted 4 lines
fs/xfs/libxfs/xfs_utf8.c new file +4 -0
+
+	major = mp->m_sb.sb_utf8version >> UNICODE_MAJ_SHIFT;
+	minor = (mp->m_sb.sb_utf8version & 0xff00) >> UNICODE_MIN_SHIFT;
+	revision = mp->m_sb.sb_utf8version & 0xff;

fs/xfs/libxfs/xfs_utf8.c line 67

Quoted 10 lines
fs/xfs/libxfs/xfs_utf8.c new file +10 -0
+
+	if (revision) {
+		xfs_warn(mp,
+		"Unicode version %d.%d.%d not supported by utf8norm.ko",
+		major, minor, revision);
+	} else {
+		xfs_warn(mp,
+		"Unicode version %d.%d not supported by utf8norm.ko",
+		major, minor);
+	}

why do you need two different print statements? Version X.Y.0 is pretty much recognisable as being the same as X.Y....

fs/xfs/libxfs/xfs_utf8.c line 69

Quoted 2 lines
fs/xfs/libxfs/xfs_utf8.c new file +2 -0
+
+	return 0;

return false;

Quoted 21 lines
fs/xfs/libxfs/xfs_utf8.c new file +1 -0
+}
fs/xfs/libxfs/xfs_utf8.h new file +14 -0
new file mode 100644
index 0000000..8a700de
@@ -0,0 +1,24 @@
+/*
+ * Copyright (c) 2014 SGI.
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License

fs/xfs/libxfs/xfs_utf8.h line 24

Quoted 10 lines
fs/xfs/libxfs/xfs_utf8.h new file +10 -0
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#ifndef XFS_UTF8_H
+#define XFS_UTF8_H
+
+extern int xfs_utf8_version_ok(struct xfs_mount *);
+
+#endif /* XFS_UTF8_H */

Do we really need a separate header file for this? fs/xfs/libxfs/xfs_shared.h was created for such one-off or limited definitions that need to be shared with userspace...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#22 lore raw

Re: [PATCH 11/16] xfs: add xfs_nameops for utf8 and utf8+casefold.

In reply to Ben Myers
Quoted 38 lines · Ben Myers

From: Olaf Weber <[email protected]>

The xfs_utf8_nameops use the nfkdi normalization when comparing filenames, and are installed if the utf8bit is set in the super block.

The xfs_utf8_ci_nameops use the nfkdicf normalization when comparing filenames, and are installed if both the utf8bit and the borgbit are set in the superblock.

Normalized filenames are not stored on disk. Normalization will fail if a filename is not valid UTF-8, in which case the filename is treated as an opaque blob.

---

[v2: updated to use utf8norm.ko module;
     compiled conditionally on CONFIG_XFS_UTF8=y;
     utf8version is now a function;
     move xfs_utf8.[ch] into libxfs. --bpm]
[v3: pass utf8version from the superblock through xfs_nameops
     instead of the max version of the normalization module. --bpm]
---
 fs/xfs/Kconfig           |   9 ++
 fs/xfs/Makefile          |   2 +
 fs/xfs/libxfs/xfs_dir2.c |   4 +-
 fs/xfs/libxfs/xfs_utf8.c | 208 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_utf8.h |   3 +
 fs/xfs/xfs_iops.c        |   2 +-
 6 files changed, 225 insertions(+), 3 deletions(-)
fs/xfs/Kconfig +0 -0
index 5d47b4d..1e8a463 100644
@@ -95,3 +95,12 @@ config XFS_DEBUG
 	  not useful unless you are debugging a particular problem.
 

fs/xfs/Kconfig config XFS_DEBUG · line 106

Quoted 10 lines
fs/xfs/Kconfig +9 -0
 	  Say N unless you are an XFS developer, or you play one on TV.
+
+config XFS_UTF8
+	bool "XFS UTF-8 support"
+	depends on XFS_FS
+	select CONFIG_UTF8_NORMALIZATION
+	help
+	  Say Y here to enable utf8 normalization support in XFS.  You
+	  will be able to mount and use filesystems created with the
+	  utf8 mkfs.xfs option.

"created with UTF8 support enabled."

fs/xfs/Makefile xfs-$(CONFIG_XFS_QUOTA) += xfs_dquot.o \ · line 118

Quoted 10 lines
fs/xfs/Makefile +2 -0
index d617999..192aaca 100644
@@ -114,6 +114,8 @@ xfs-$(CONFIG_XFS_QUOTA)		+= xfs_dquot.o \
 				   xfs_qm.o \
 				   xfs_quotaops.o
 
+xfs-$(CONFIG_XFS_UTF8)		+= libxfs/xfs_utf8.o
+

libxfs definitions come first. Also, please use the same prefixing syntax that the other libxfs rules use.

Quoted 8 lines
fs/xfs/Makefile +0 -0
 # xfs_rtbitmap is shared with libxfs
 xfs-$(CONFIG_XFS_RT)		+= xfs_rtalloc.o
 
fs/xfs/libxfs/xfs_dir2.c +0 -0
index 2c89211..9cfbd6b 100644

fs/xfs/libxfs/xfs_dir2.c xfs_da_mount( · line 171

Quoted 9 lines
fs/xfs/libxfs/xfs_dir2.c +2 -2
@@ -165,9 +165,9 @@ xfs_da_mount(
 		/* XXX these are replaced in the next patch need
 		   to do some kind of reordering here */
 		if (xfs_sb_version_hasasciici(&mp->m_sb))
-			mp->m_dirnameops = &xfs_ascii_ci_nameops;
+			mp->m_dirnameops = &xfs_utf8_ci_nameops;
 		else
-			mp->m_dirnameops = &xfs_default_nameops;
+			mp->m_dirnameops = &xfs_utf8_nameops;
 #else

xfs_sb_version_hasasciici()? The overloading of the asciici bit is still used for the utf8 CI functionality? Please fix this for the next version of the patchset.

Quoted 7 lines
fs/xfs/libxfs/xfs_dir2.c +0 -0
 		xfs_warn(mp,
 "Recompile XFS with CONFIG_XFS_UTF8 to mount this filesystem");
fs/xfs/libxfs/xfs_utf8.c +0 -0
index 7e63111..1e75299 100644

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 74

Quoted 7 lines
fs/xfs/libxfs/xfs_utf8.c +4 -0
@@ -68,3 +68,211 @@ xfs_utf8_version_ok(
 
 	return 0;
 }
+
+/*
+ * xfs nameops using nfkdi
+ */

Remind me again what nfkdi means? I I can't remember the details after a week or two, then perhaps better explanitory comments are needed in the code?

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 80

Quoted 5 lines
fs/xfs/libxfs/xfs_utf8.c +5 -0
+static xfs_dahash_t
+xfs_utf8_hashname(
+	const unsigned char *name,
+	int len,
+	unsigned int sb_utf8version)

Please use the same indentation levels for the declartions. i.e

const unsigned char *name,
int len,
unsigned int sb_utf8version)

Can you go through all the XFS code and make sure this is done?

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 85

Quoted 5 lines
fs/xfs/libxfs/xfs_utf8.c +5 -0
+{
+	utf8data_t	nfkdi;
+	struct utf8cursor u8c;
+	xfs_dahash_t	hash;
+	int		val;

And these shold line up, too.

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 88

Quoted 3 lines
fs/xfs/libxfs/xfs_utf8.c +3 -0
+
+	nfkdi = utf8nfkdi(sb_utf8version);
+	hash = 0;

initialise at declaration.

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 89

Quoted 2 lines
fs/xfs/libxfs/xfs_utf8.c +2 -0
+	if (utf8ncursor(&u8c, nfkdi, name, len) < 0)
+		goto blob;

Still has the "invalid binary blob" issue.

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 101

Quoted 11 lines
fs/xfs/libxfs/xfs_utf8.c +11 -0
+	while ((val = utf8byte(&u8c)) > 0)
+		hash = val ^ rol32(hash, 7);
+	/* In case of error treat the name as a binary blob. */
+	if (val == 0)
+		return hash;
+blob:
+	return xfs_da_hashname(name, len);
+}
+
+static int
+xfs_utf8_normhash(

More commments needed explaining what is going on.

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 110

Quoted 9 lines
fs/xfs/libxfs/xfs_utf8.c +9 -0
+	struct xfs_da_args *args)
+{
+	utf8data_t	nfkdi;
+	struct utf8cursor u8c;
+	unsigned char	*norm;
+	ssize_t		normlen;
+	int		c;
+	unsigned int	sb_utf8version =
+		args->dp->i_mount->m_sb.sb_utf8version;

Urk. Initialise on a separate line.

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 115

Quoted 5 lines
fs/xfs/libxfs/xfs_utf8.c +5 -0
+
+	nfkdi = utf8nfkdi(sb_utf8version);
+	/* Failure to normalize is treated as a blob. */
+	if ((normlen = utf8nlen(nfkdi, args->name, args->namelen)) < 0)
+		goto blob;

No assignments in logic statements, please.

	normlen = utf8nlen(nfkdi, args->name, args->namelen);
	if (normlen < 0)

This is all through the code - can you please go through and fix up all the patches to remove this pattern? checkpatch might be helpful here....

As it is, still has the invalid binary blob issue.

fs/xfs/libxfs/xfs_utf8.c xfs_utf8_version_ok( · line 119

Quoted 4 lines
fs/xfs/libxfs/xfs_utf8.c +4 -0
+	if (utf8ncursor(&u8c, nfkdi, args->name, args->namelen) < 0)
+		goto blob;
+	if (!(norm = kmem_alloc(normlen + 1, KM_NOFS|KM_MAYFAIL)))
+		return -ENOMEM;

Urk.

So, what happens if this memory allocation fails in the middle of a create transaction?

(Hint: transaction is dirty at this point in time)

The rest of the code in this patch has similar issues to what I've already pointed out.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#14 lore raw

Re: [PATCH 08/16] xfs: change interface of xfs_nameops.hashname

In reply to Ben Myers
Quoted 10 lines · Ben Myers

From: Olaf Weber <[email protected]>

With the introduction of the xfs_nameops.normhash callout, all uses of the hashname callout now occur in places where an xfs_name structure must be explicitly created just to match the parameter passing convention of this callout. Change the arguments to a const unsigned char * and int instead.

[v2: pass a 3rd argument for sb_utf8version to hashname. --bpm]

So now I've looked at most of the rest of the patch set, I think this is the wrong thing to do. I see no reason apart from "it's less typing" to drop the use of the xfs-name structure, but it removes a key piece of documentation from the code. i.e. that the name/namelen are an inseparable tuple and cannot be separated. Indeed, lots of the utf8 xfs code declares norm/normlen tuples on the stack for temporary use, so really this comes down to a matter of taste.

And in that matter, I'd prefer that we keep the existing name abstaction and propagate it into the new code rather than the other way around.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw

Re: [PATCH 05/16] xfs: return the first match during case-insensitive lookup.

In reply to Ben Myers
Quoted 9 lines · Ben Myers

From: Olaf Weber <[email protected]>

Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.

This is really dependent on whether we want to support mixed CI/non-CI filesystems, yes? i.e. if we want to support mixed case setups, then we need to keep the code as it stands? What is the downside of keeping the code unchaged and our options open?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#26 lore raw

Re: [PATCH 13/16] xfs: implement demand load of utf8norm.ko

In reply to Christoph Hellwig
Quoted 17 lines · Christoph Hellwig
Quoted 17 lines
fs/xfs/libxfs/xfs_utf8.c +17 -0
+int
+xfs_init_utf8_module(struct xfs_mount	*mp)
+{
+	request_module("utf8norm");
+
+	spin_lock(&utf8norm_lock);
+	if (utf8norm_initialized) {
+		spin_unlock(&utf8norm_lock);
+		return 0;
+	}
+
+	utf8version_is_supported_func = symbol_get(utf8version_is_supported);
+	if (!utf8version_is_supported_func)
+		goto error;
+
+	utf8nfkdi_func = symbol_get(utf8nfkdi);
+	if (!utf8nfkdi_func)
Quoted 8 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+		goto error;

Please export a structure with a function pointes so that we just need a single symbol_get call. I'd have to look up how symbol_get works, but unless there's something that speaks against this it might be simpler to than just do a symbol_get per mounst structure that uses utf8 and can point to that structure so that we don't have to add additional reference counting infrastructure around it.

Sure, sounds good. I believe I've seen the approach you are suggesting used elsewhere and it works fine.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#18 lore raw

Re: [PATCH 09/16] xfs: add a superblock feature bit to indicate UTF-8 support.

In reply to Dave Chinner
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Ben Myers

From: Olaf Weber <[email protected]>

When UTF-8 support is enabled, the xfs_dir_ci_inode_operations must be installed. Add xfs_sb_version_hasci(), which tests both the borgbit and the utf8bit, and returns true if at least one of them is set. Replace calls to xfs_sb_version_hasasciici() as needed.

---
 fs/xfs/libxfs/xfs_sb.h | 24 +++++++++++++++++++++++-
 fs/xfs/xfs_fs.h        |  1 +
 fs/xfs/xfs_fsops.c     |  4 +++-
 fs/xfs/xfs_iops.c      |  4 ++--
 4 files changed, 29 insertions(+), 4 deletions(-)
Quoted 12 lines

fs/xfs/libxfs/xfs_sb.h struct xfs_trans; · line 74

Quoted 10 lines
fs/xfs/libxfs/xfs_sb.h +1 -0
index 2e73970..525eacb 100644
@@ -70,6 +70,7 @@ struct xfs_trans;
 #define XFS_SB_VERSION2_RESERVED4BIT	0x00000004
 #define XFS_SB_VERSION2_ATTR2BIT	0x00000008	/* Inline attr rework */
 #define XFS_SB_VERSION2_PARENTBIT	0x00000010	/* parent pointers */
+#define XFS_SB_VERSION2_UTF8BIT		0x00000020      /* utf8 names */
 #define XFS_SB_VERSION2_PROJID32BIT	0x00000080	/* 32 bit project id */

Can you explain why this bit is safe to use?

I believe Olaf chose this value to match what was used in Barry's implementation.

Quoted 4 lines

I don't recall why
XFS_SB_VERSION2_PROJID32BIT skipped several bits because there aren't any comments explaining why that value was chosen. Adding a comment about the 0x00000040 bit at the same time would be useful.

I'm not sure why we skipped. I'll see what I can find in mail archives.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#8 lore raw

Re: [PATCH 05/16] xfs: return the first match during case-insensitive lookup.

In reply to Dave Chinner
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Ben Myers

From: Olaf Weber <[email protected]>

Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.

Quoted 5 lines
Quoted 1 line

This is really dependent on whether we want to support mixed CI/non-CI filesystems, yes? i.e. if we want to support mixed case setups, then we need to keep the code as it stands?

It depends upon what semantics you decide are correct in the mixed case. This is just one solution.

Quoted 2 lines

What is the
downside of keeping the code unchaged and our options open?

The code that is being removed here was for the case when you could have multiple filenames that match for a lookup, which was only possible when the ascii-ci bit was implemented as a mount option. The mount option was never merged, and the ascii-ci bit is implemented at mkfs and is not changeable over the lifetime of the filesystem. So today this code is not doing us much good. Maybe there are performance implications where you have a hash collision, but mostly I think it makes sense to remove this code from a maintainability standpoint.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw

Re: [PATCH 05/16] xfs: return the first match during case-insensitive lookup.

Quoted 10 lines · Ben Myers
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Ben Myers

From: Olaf Weber <[email protected]>

Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

This is really dependent on whether we want to support mixed CI/non-CI filesystems, yes? i.e. if we want to support mixed case setups, then we need to keep the code as it stands?

It depends upon what semantics you decide are correct in the mixed case. This is just one solution.

Ok, so we need this code or somethign very similar to support mixed case filesystems. Can you tell us what the other possible solutions and semantics have been considered?

Quoted 7 lines
Quoted 2 lines

What is the downside of keeping the code unchaged and our options open?

The code that is being removed here was for the case when you could have multiple filenames that match for a lookup, which was only possible when the ascii-ci bit was implemented as a mount option.

Yes, I know what the code did - it allowed us to support mixed case ascii-ci filesystems. All you've said is "if we remove mixed case support the code is cleaner" but not addressed the issue at hand.

I'll try asking the same question a different way: if we keep this code, will it work for mixed case unicode filesystem or do we have to re-implement mixed case matching differently?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#10 lore raw

Re: [PATCH 05/16] xfs: return the first match during case-insensitive lookup.

Dave,

Quoted 17 lines · Dave Chinner
Quoted 16 lines · Ben Myers
Quoted 14 lines · Dave Chinner
Quoted 9 lines · Ben Myers

From: Olaf Weber <[email protected]>

Change the XFS case-insensitive lookup code to return the first match found, even if it is not an exact match. Whether a filesystem uses case-insensitive lookups is determined by a superblock bit set during filesystem creation. This means that normal use cannot create two files that both match the same filename.

This is really dependent on whether we want to support mixed CI/non-CI filesystems, yes? i.e. if we want to support mixed case setups, then we need to keep the code as it stands?

Quoted 6 lines
Quoted 2 lines

It depends upon what semantics you decide are correct in the mixed case. This is just one solution.

Ok, so we need this code or somethign very similar to support mixed case filesystems. Can you tell us what the other possible solutions and semantics have been considered?

There was some discussion of this in the v2 posting of this rfc.

http://marc.info/?l=linux-xfs&m=141176024430150&w=2

Olaf's "readme" example at the above link is a pretty good example of what we're facing. And I don't have a good answer for which file to open. So for now we're just going for the cleanest solution.

Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

What is the downside of keeping the code unchaged and our options open?

The code that is being removed here was for the case when you could have multiple filenames that match for a lookup, which was only possible when the ascii-ci bit was implemented as a mount option.

Yes, I know what the code did - it allowed us to support mixed case ascii-ci filesystems. All you've said is "if we remove mixed case support the code is cleaner" but not addressed the issue at hand.

I also tried to explain that as the codebase stands today, removal of this code does not represent a loss of functionality. It is dead code.

Quoted 3 lines

I'll try asking the same question a different way: if we keep this code, will it work for mixed case unicode filesystem or do we have to re-implement mixed case matching differently?

If you definately want to keep this code around I'll look into this, but right now I don't have plans to extend the patchset to support mixed case insensitivity in a single filesystem.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#15 lore raw

Re: [PATCH 08/16] xfs: change interface of xfs_nameops.hashname

In reply to Dave Chinner
Quoted 20 lines · Dave Chinner
Quoted 10 lines · Ben Myers

From: Olaf Weber <[email protected]>

With the introduction of the xfs_nameops.normhash callout, all uses of the hashname callout now occur in places where an xfs_name structure must be explicitly created just to match the parameter passing convention of this callout. Change the arguments to a const unsigned char * and int instead.

[v2: pass a 3rd argument for sb_utf8version to hashname. --bpm]

So now I've looked at most of the rest of the patch set, I think this is the wrong thing to do. I see no reason apart from "it's less typing" to drop the use of the xfs-name structure, but it removes a key piece of documentation from the code. i.e. that the name/namelen are an inseparable tuple and cannot be separated. Indeed, lots of the utf8 xfs code declares norm/normlen tuples on the stack for temporary use, so really this comes down to a matter of taste.

Quoted 3 lines

And in that matter, I'd prefer that we keep the existing name abstaction and propagate it into the new code rather than the other way around.

Does something like this suit you?

struct xfs_name {
	const unsigned char	*name;
	int			len;
	int			type;
	__uint32_t		utf8version;
};

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs