Skip to contentSkip to conversation

dedupe smb unicode files

PATCH In review [email protected] 2 participants 6 messages Aug 17, 2023 – Aug 19, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v5 cover dedupe smb unicode files

From: "Dr. David Alan Gilbert" <[email protected]>

The smb client and server code have (mostly) duplicated code for unicode manipulation, in particular upper case handling.

Flatten this lot into shared code.

There's some code that's slightly different between the two, and I've not attempted to share that - this should be strictly a no behaviour change set.

In addition, the same tables and code are shared in jfs, however there's very little testing available for the unicode in there, so just share the raw data tables.

I suspect there's more UCS-2 code that can be shared, in the NLS code and in the UCS-2 code used by the EFI interfaces.

Lightly tested with a module and a monolithic build, and just mounting itself.

This dupe was found using PMD:
https://pmd.github.io/pmd/pmd_userdocs_cpd.html

Dave

Version 5
Add some (wchar_t *) casts to keep sparse
happy, as spotted by kernel test robot

Dr. David Alan Gilbert (4):
fs/smb: Remove unicode 'lower' tables
fs/smb: Swing unicode common code from smb->NLS
fs/smb/client: Use common code in client
fs/jfs: Use common ucs2 upper case table

 fs/jfs/Kconfig                                |   1 +
 fs/jfs/Makefile                               |   2 +-
 fs/jfs/jfs_unicode.h                          |  17 +-
 fs/jfs/jfs_uniupr.c                           | 121 -------
 fs/nls/Kconfig                                |   8 +
 fs/nls/Makefile                               |   1 +
 fs/nls/nls_ucs2_data.h                        |  15 +
 .../server/uniupr.h => nls/nls_ucs2_utils.c}  | 156 +--------
 fs/nls/nls_ucs2_utils.h                       | 285 +++++++++++++++
 fs/smb/client/Kconfig                         |   1 +
 fs/smb/client/cifs_unicode.c                  |   1 -
 fs/smb/client/cifs_unicode.h                  | 330 +-----------------
 fs/smb/client/cifs_uniupr.h                   | 239 -------------
 fs/smb/client/smb2pdu.c                       |   4 +-
 fs/smb/server/Kconfig                         |   1 +
 fs/smb/server/unicode.c                       |   1 -
 fs/smb/server/unicode.h                       | 325 +----------------
 17 files changed, 342 insertions(+), 1166 deletions(-)
 delete mode 100644 fs/jfs/jfs_uniupr.c
 create mode 100644 fs/nls/nls_ucs2_data.h
 rename fs/{smb/server/uniupr.h => nls/nls_ucs2_utils.c} (50%)
 create mode 100644 fs/nls/nls_ucs2_utils.h
 delete mode 100644 fs/smb/client/cifs_uniupr.h
Signature · 1 line
2.41.0
#1 lore raw

PATCH v5 1/4 fs/smb: Remove unicode 'lower' tables

From: "Dr. David Alan Gilbert" <[email protected]>

The unicode glue in smb/*/..uniupr.h has a section guarded by 'ifndef UNIUPR_NOLOWER' - but that's always defined in smb/*/..unicode.h. Nuke those tables.

---
 fs/smb/client/cifs_unicode.h |  50 --------------
 fs/smb/client/cifs_uniupr.h  | 116 -------------------------------
 fs/smb/server/unicode.h      |  48 -------------
 fs/smb/server/uniupr.h       | 129 -----------------------------------
 4 files changed, 343 deletions(-)
fs/smb/client/cifs_unicode.h +0 -50
index 80b3d845419f..33b651def482 100644
@@ -22,8 +22,6 @@
 #include <linux/types.h>
 #include <linux/nls.h>
 
-#define  UNIUPR_NOLOWER		/* Example to not expand lower case tables */
-
 /*
  * Windows maps these to the user defined 16 bit Unicode range since they are
  * reserved symbols (along with \ and /), otherwise illegal to store
@@ -84,11 +82,6 @@ extern signed char CifsUniUpperTable[512];
 extern const struct UniCaseRange CifsUniUpperRange[];
 #endif				/* UNIUPR_NOUPPER */
 
-#ifndef UNIUPR_NOLOWER
-extern signed char CifsUniLowerTable[512];
-extern const struct UniCaseRange CifsUniLowerRange[];
-#endif				/* UNIUPR_NOLOWER */
-
 #ifdef __KERNEL__
 int cifs_from_utf16(char *to, const __le16 *from, int tolen, int fromlen,
 		    const struct nls_table *cp, int map_type);
@@ -358,47 +351,4 @@ UniStrupr(register __le16 *upin)
 }
 #endif				/* UNIUPR_NOUPPER */
 
-#ifndef UNIUPR_NOLOWER
-/*
- * UniTolower:  Convert a unicode character to lower case
- */
-static inline wchar_t
-UniTolower(register wchar_t uc)
-{
-	register const struct UniCaseRange *rp;
-
-	if (uc < sizeof(CifsUniLowerTable)) {
-		/* Latin characters */
-		return uc + CifsUniLowerTable[uc];	/* Use base tables */
-	} else {
-		rp = CifsUniLowerRange;	/* Use range tables */
-		while (rp->start) {
-			if (uc < rp->start)	/* Before start of range */
-				return uc;	/* Uppercase = input */
-			if (uc <= rp->end)	/* In range */
-				return uc + rp->table[uc - rp->start];
-			rp++;	/* Try next range */
-		}
-	}
-	return uc;		/* Past last range */
-}
-
-/*
- * UniStrlwr:  Lower case a unicode string
- */
-static inline wchar_t *
-UniStrlwr(register wchar_t *upin)
-{
-	register wchar_t *up;
-
-	up = upin;
-	while (*up) {		/* For all characters */
-		*up = UniTolower(*up);
-		up++;
-	}
-	return upin;		/* Return input pointer */
-}
-
-#endif
-
 #endif /* _CIFS_UNICODE_H */
fs/smb/client/cifs_uniupr.h +0 -116

119 lines not on this page · raw message

fs/smb/server/unicode.h +0 -48

64 lines not on this page · raw message

fs/smb/server/uniupr.h +0 -129

133 lines not on this page · raw message

Signature · 1 line
2.41.0
#2 lore raw

PATCH v5 2/4 fs/smb: Swing unicode common code from smb->NLS

In reply to [email protected]

From: "Dr. David Alan Gilbert" <[email protected]>

Swing most of the inline functions and unicode tables into nls from the copy in smb/server. This is UCS-2 rather than most of the rest of the code in NLS, but it currently seems like the best place for it.

The actual unicode.c implementations vary much more between server and client so they're unmoved.

---
 fs/nls/Kconfig                                |   8 +
 fs/nls/Makefile                               |   1 +
 .../server/uniupr.h => nls/nls_ucs2_utils.c}  |  27 +-
 fs/nls/nls_ucs2_utils.h                       | 297 ++++++++++++++++++
 fs/smb/server/Kconfig                         |   1 +
 fs/smb/server/unicode.c                       |   1 -
 fs/smb/server/unicode.h                       | 277 +---------------
 7 files changed, 328 insertions(+), 284 deletions(-)
 rename fs/{smb/server/uniupr.h => nls/nls_ucs2_utils.c} (91%)
 create mode 100644 fs/nls/nls_ucs2_utils.h
fs/nls/Kconfig +8 -0
index c7857e36adbb..a0d0e2f7ec83 100644
@@ -617,4 +617,12 @@ config NLS_UTF8
 	  input/output character sets. Say Y here for the UTF-8 encoding of
 	  the Unicode/ISO9646 universal character set.
 
+config NLS_UCS2_UTILS
+	tristate "NLS UCS-2 UTILS"
+	help
+	  Set of older UCS-2 conversion utilities and tables used by some
+	  filesystems including SMB/CIFS.  This includes upper case conversion
+	  tables. This will automatically be selected when the filesystem
+	  that uses it is selected.
+
 endif # NLS
fs/nls/Makefile +1 -0
index ac54db297128..5062c699d041 100644
@@ -54,3 +54,4 @@ obj-$(CONFIG_NLS_MAC_INUIT)     += mac-inuit.o
 obj-$(CONFIG_NLS_MAC_ROMANIAN)  += mac-romanian.o
 obj-$(CONFIG_NLS_MAC_ROMAN)     += mac-roman.o
 obj-$(CONFIG_NLS_MAC_TURKISH)   += mac-turkish.o
+obj-$(CONFIG_NLS_UCS2_UTILS)		+= nls_ucs2_utils.o
fs/nls/nls_ucs2_utils.c renamed from fs/smb/server/uniupr.h +17 -10
similarity index 91%
rename from fs/smb/server/uniupr.h
rename to fs/nls/nls_ucs2_utils.c
index d09c585f20c9..a69781c54dd8 100644
@@ -1,19 +1,27 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
+// SPDX-License-Identifier: GPL-2.0-or-later
 /*
  *   Some of the source code in this file came from fs/cifs/uniupr.h
  *   Copyright (c) International Business Machines  Corp., 2000,2002
  *
- * uniupr.h - Unicode compressed case ranges
+ *   Some of the source code in this file came from fs/cifs/cifs_unicode.c
+ *
+ *   Copyright (c) International Business Machines  Corp., 2000,2009
+ *   Modified by Steve French ([email protected])
+ *   Modified by Namjae Jeon ([email protected])
  *
  */
-#ifndef __KSMBD_UNIUPR_H
-#define __KSMBD_UNIUPR_H
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <asm/unaligned.h>
+#include "nls_ucs2_utils.h"
+
+MODULE_LICENSE("GPL");
 
-#ifndef UNIUPR_NOUPPER
 /*
  * Latin upper case
  */
-signed char SmbUniUpperTable[512] = {
+signed char NlsUniUpperTable[512] = {
 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,	/* 000-00f */
 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,	/* 010-01f */
 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,	/* 020-02f */
@@ -51,6 +59,7 @@ signed char SmbUniUpperTable[512] = {
 	0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1,	/* 1e0-1ef */
 	0, 0, -1, -2, 0, -1, 0, 0, 0, -1, 0, -1, 0, -1, 0, -1,	/* 1f0-1ff */
 };
+EXPORT_SYMBOL_GPL(NlsUniUpperTable);
 
 /* Upper case range - Greek */
 static signed char UniCaseRangeU03a0[47] = {
@@ -126,7 +135,7 @@ static signed char UniCaseRangeUff40[27] = {
 /*
  * Upper Case Range
  */
-const struct UniCaseRange SmbUniUpperRange[] = {
+const struct UniCaseRange NlsUniUpperRange[] = {
 	{0x03a0, 0x03ce, UniCaseRangeU03a0},
 	{0x0430, 0x045f, UniCaseRangeU0430},
 	{0x0490, 0x04cc, UniCaseRangeU0490},
@@ -134,6 +143,4 @@ const struct UniCaseRange SmbUniUpperRange[] = {
 	{0xff40, 0xff5a, UniCaseRangeUff40},
 	{0}
 };
-#endif
-
-#endif /* __KSMBD_UNIUPR_H */
+EXPORT_SYMBOL_GPL(NlsUniUpperRange);
fs/nls/nls_ucs2_utils.h new file +297 -0

297 lines not on this page · raw message

fs/smb/server/Kconfig +1 -0
index 7055cb5d2880..793151ddd60e 100644
@@ -5,6 +5,7 @@ config SMB_SERVER
 	depends on FILE_LOCKING
 	select NLS
 	select NLS_UTF8
+	select NLS_UCS2_UTILS
 	select CRYPTO
 	select CRYPTO_MD5
 	select CRYPTO_HMAC
fs/smb/server/unicode.c +0 -1
index 9ae676906ed3..393dd4a7432b 100644
@@ -11,7 +11,6 @@
 #include <asm/unaligned.h>
 #include "glob.h"
 #include "unicode.h"
-#include "uniupr.h"
 #include "smb_common.h"
 
 /*
fs/smb/server/unicode.h +4 -273

291 lines not on this page · raw message

Signature · 1 line
2.41.0
#3 lore raw

PATCH v5 3/4 fs/smb/client: Use common code in client

In reply to [email protected]

From: "Dr. David Alan Gilbert" <[email protected]>

Now we've got the common code, use it for the client as well. Note there's a change here where we're using the server version of UniStrcat now which had different types (__le16 vs wchar_t) but it's not interpreting the value other than checking for 0, however we do need casts to keep sparse happy.

---
 fs/smb/client/Kconfig        |   1 +
 fs/smb/client/cifs_unicode.c |   1 -
 fs/smb/client/cifs_unicode.h | 280 +----------------------------------
 fs/smb/client/cifs_uniupr.h  | 123 ---------------
 fs/smb/client/smb2pdu.c      |   4 +-
 5 files changed, 4 insertions(+), 405 deletions(-)
 delete mode 100644 fs/smb/client/cifs_uniupr.h
fs/smb/client/Kconfig +1 -0
index 4c0d53bf931a..2927bd174a88 100644
@@ -3,6 +3,7 @@ config CIFS
 	tristate "SMB3 and CIFS support (advanced network filesystem)"
 	depends on INET
 	select NLS
+	select NLS_UCS2_UTILS
 	select CRYPTO
 	select CRYPTO_MD5
 	select CRYPTO_SHA256
fs/smb/client/cifs_unicode.c +0 -1
index e7582dd79179..79d99a913944 100644
@@ -8,7 +8,6 @@
 #include <linux/slab.h>
 #include "cifs_fs_sb.h"
 #include "cifs_unicode.h"
-#include "cifs_uniupr.h"
 #include "cifspdu.h"
 #include "cifsglob.h"
 #include "cifs_debug.h"
fs/smb/client/cifs_unicode.h +1 -279

296 lines not on this page · raw message

fs/smb/client/cifs_uniupr.h deleted +0 -123

123 lines not on this page · raw message

fs/smb/client/smb2pdu.c +2 -2
index a457f07f820d..bbc45745bbfc 100644
@@ -2570,8 +2570,8 @@ alloc_path_with_tree_prefix(__le16 **out_path, int *out_size, int *out_len,
 
 	/* Do not append the separator if the path is empty */
 	if (path[0] != cpu_to_le16(0x0000)) {
-		UniStrcat(*out_path, sep);
-		UniStrcat(*out_path, path);
+		UniStrcat((wchar_t *)*out_path, (wchar_t *)sep);
+		UniStrcat((wchar_t *)*out_path, (wchar_t *)path);
 	}
 
 	unload_nls(cp);
Signature · 1 line
2.41.0
#4 lore raw

PATCH v5 4/4 fs/jfs: Use common ucs2 upper case table

In reply to [email protected]

From: "Dr. David Alan Gilbert" <[email protected]>

Use the UCS-2 upper case tables from nls, that are shared with smb.

This code in JFS is hard to test, so we're only reusing the same tables (which are identical), not trying to reuse the rest of the helper functions.

---
 fs/jfs/Kconfig          |   1 +
 fs/jfs/Makefile         |   2 +-
 fs/jfs/jfs_unicode.h    |  17 ++----
 fs/jfs/jfs_uniupr.c     | 121 ----------------------------------------
 fs/nls/nls_ucs2_data.h  |  15 +++++
 fs/nls/nls_ucs2_utils.h |  14 +----
 6 files changed, 23 insertions(+), 147 deletions(-)
 delete mode 100644 fs/jfs/jfs_uniupr.c
 create mode 100644 fs/nls/nls_ucs2_data.h
fs/jfs/Kconfig +1 -0
index 51e856f0e4b8..eab2f2d2291f 100644
@@ -2,6 +2,7 @@
 config JFS_FS
 	tristate "JFS filesystem support"
 	select NLS
+	select NLS_UCS2_UTILS
 	select CRC32
 	select LEGACY_DIRECT_IO
 	help
fs/jfs/Makefile +1 -1
index 7156d2c218c7..b769bbf8bdc2 100644
@@ -9,7 +9,7 @@ jfs-y    := super.o file.o inode.o namei.o jfs_mount.o jfs_umount.o \
 	    jfs_xtree.o jfs_imap.o jfs_debug.o jfs_dmap.o \
 	    jfs_unicode.o jfs_dtree.o jfs_inode.o jfs_discard.o \
 	    jfs_extent.o symlink.o jfs_metapage.o \
-	    jfs_logmgr.o jfs_txnmgr.o jfs_uniupr.o \
+	    jfs_logmgr.o jfs_txnmgr.o \
 	    resize.o xattr.o ioctl.o
 
 jfs-$(CONFIG_JFS_POSIX_ACL) += acl.o
fs/jfs/jfs_unicode.h +5 -12
index 9db62d047daa..b6a78d4aef1b 100644
@@ -8,16 +8,9 @@
 
 #include <linux/slab.h>
 #include <asm/byteorder.h>
+#include "../nls/nls_ucs2_data.h"
 #include "jfs_types.h"
 
-typedef struct {
-	wchar_t start;
-	wchar_t end;
-	signed char *table;
-} UNICASERANGE;
-
-extern signed char UniUpperTable[512];
-extern UNICASERANGE UniUpperRange[];
 extern int get_UCSname(struct component_name *, struct dentry *);
 extern int jfs_strfromUCS_le(char *, const __le16 *, int, struct nls_table *);
 
@@ -107,12 +100,12 @@ static inline wchar_t *UniStrncpy_from_le(wchar_t * ucs1, const __le16 * ucs2,
  */
 static inline wchar_t UniToupper(wchar_t uc)
 {
-	UNICASERANGE *rp;
+	const struct UniCaseRange *rp;
 
-	if (uc < sizeof(UniUpperTable)) {	/* Latin characters */
-		return uc + UniUpperTable[uc];	/* Use base tables */
+	if (uc < sizeof(NlsUniUpperTable)) {	/* Latin characters */
+		return uc + NlsUniUpperTable[uc];	/* Use base tables */
 	} else {
-		rp = UniUpperRange;	/* Use range tables */
+		rp = NlsUniUpperRange;	/* Use range tables */
 		while (rp->start) {
 			if (uc < rp->start)	/* Before start of range */
 				return uc;	/* Uppercase = input */
fs/jfs/jfs_uniupr.c deleted +0 -121

121 lines not on this page · raw message

fs/nls/nls_ucs2_data.h new file +15 -0
new file mode 100644
index 000000000000..1f454dc0f4e0
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef _NLS_UCS2_DATA_H
+#define _NLS_UCS2_DATA_H
+
+struct UniCaseRange {
+	wchar_t start;
+	wchar_t end;
+	signed char *table;
+};
+
+extern signed char NlsUniUpperTable[512];
+extern const struct UniCaseRange NlsUniUpperRange[];
+
+#endif /* _NLS_UCS2_DATA_H */
fs/nls/nls_ucs2_utils.h +1 -13
index 3500596ea993..ef18d30db1d0 100644
@@ -26,6 +26,7 @@
 #include <linux/types.h>
 #include <linux/nls.h>
 #include <linux/unicode.h>
+#include "nls_ucs2_data.h"
 
 /*
  * Windows maps these to the user defined 16 bit Unicode range since they are
@@ -40,19 +41,6 @@
 #define UNI_PIPE        ((__u16)('|' + 0xF000))
 #define UNI_SLASH       ((__u16)('\\' + 0xF000))
 
-#ifndef	UNICASERANGE_DEFINED
-struct UniCaseRange {
-	wchar_t start;
-	wchar_t end;
-	signed char *table;
-};
-#endif				/* UNICASERANGE_DEFINED */
-
-#ifndef UNIUPR_NOUPPER
-extern signed char NlsUniUpperTable[512];
-extern const struct UniCaseRange NlsUniUpperRange[];
-#endif				/* UNIUPR_NOUPPER */
-
 /*
  * UniStrcat:  Concatenate the second string to the first
  *
Signature · 1 line
2.41.0
#5 lore raw
In reply to [email protected]

tentatively merged to cifs-2.6.git for-next pending additional testing

Let me know if anyone wants to add RB or Acked-by to any of these or spots any issues

Quoted 64 lines · On Wed, Aug 16, 2023 at 7:22 PM <[email protected]> wrote:

From: "Dr. David Alan Gilbert" <[email protected]>

The smb client and server code have (mostly) duplicated code for unicode manipulation, in particular upper case handling.

Flatten this lot into shared code.

There's some code that's slightly different between the two, and I've not attempted to share that - this should be strictly a no behaviour change set.

In addition, the same tables and code are shared in jfs, however there's very little testing available for the unicode in there, so just share the raw data tables.

I suspect there's more UCS-2 code that can be shared, in the NLS code and in the UCS-2 code used by the EFI interfaces.

Lightly tested with a module and a monolithic build, and just mounting itself.

This dupe was found using PMD:
https://pmd.github.io/pmd/pmd_userdocs_cpd.html

Dave

Version 5
Add some (wchar_t *) casts to keep sparse
happy, as spotted by kernel test robot

Dr. David Alan Gilbert (4):
fs/smb: Remove unicode 'lower' tables
fs/smb: Swing unicode common code from smb->NLS
fs/smb/client: Use common code in client
fs/jfs: Use common ucs2 upper case table

 fs/jfs/Kconfig                                |   1 +
 fs/jfs/Makefile                               |   2 +-
 fs/jfs/jfs_unicode.h                          |  17 +-
 fs/jfs/jfs_uniupr.c                           | 121 -------
 fs/nls/Kconfig                                |   8 +
 fs/nls/Makefile                               |   1 +
 fs/nls/nls_ucs2_data.h                        |  15 +
 .../server/uniupr.h => nls/nls_ucs2_utils.c}  | 156 +--------
 fs/nls/nls_ucs2_utils.h                       | 285 +++++++++++++++
 fs/smb/client/Kconfig                         |   1 +
 fs/smb/client/cifs_unicode.c                  |   1 -
 fs/smb/client/cifs_unicode.h                  | 330 +-----------------
 fs/smb/client/cifs_uniupr.h                   | 239 -------------
 fs/smb/client/smb2pdu.c                       |   4 +-
 fs/smb/server/Kconfig                         |   1 +
 fs/smb/server/unicode.c                       |   1 -
 fs/smb/server/unicode.h                       | 325 +----------------
 17 files changed, 342 insertions(+), 1166 deletions(-)
 delete mode 100644 fs/jfs/jfs_uniupr.c
 create mode 100644 fs/nls/nls_ucs2_data.h
 rename fs/{smb/server/uniupr.h => nls/nls_ucs2_utils.c} (50%)
 create mode 100644 fs/nls/nls_ucs2_utils.h
 delete mode 100644 fs/smb/client/cifs_uniupr.h

--
2.41.0

Signature · 3 lines
Thanks,

Steve