dedupe smb unicode files
Subsystems (3)
v5 cover dedupe smb unicode files
From: "Dr. David Alan Gilbert" <[email protected]>
The smb client and server code have (mostly) duplicated code for unicode manipulation, in particular upper case handling.
Flatten this lot into shared code.
There's some code that's slightly different between the two, and I've not attempted to share that - this should be strictly a no behaviour change set.
In addition, the same tables and code are shared in jfs, however there's very little testing available for the unicode in there, so just share the raw data tables.
I suspect there's more UCS-2 code that can be shared, in the NLS code and in the UCS-2 code used by the EFI interfaces.
Lightly tested with a module and a monolithic build, and just mounting itself.
This dupe was found using PMD:
https://pmd.github.io/pmd/pmd_userdocs_cpd.html
Dave
Version 5
Add some (wchar_t *) casts to keep sparse
happy, as spotted by kernel test robot
Dr. David Alan Gilbert (4):
fs/smb: Remove unicode 'lower' tables
fs/smb: Swing unicode common code from smb->NLS
fs/smb/client: Use common code in client
fs/jfs: Use common ucs2 upper case table
fs/jfs/Kconfig | 1 +
fs/jfs/Makefile | 2 +-
fs/jfs/jfs_unicode.h | 17 +-
fs/jfs/jfs_uniupr.c | 121 -------
fs/nls/Kconfig | 8 +
fs/nls/Makefile | 1 +
fs/nls/nls_ucs2_data.h | 15 +
.../server/uniupr.h => nls/nls_ucs2_utils.c} | 156 +--------
fs/nls/nls_ucs2_utils.h | 285 +++++++++++++++
fs/smb/client/Kconfig | 1 +
fs/smb/client/cifs_unicode.c | 1 -
fs/smb/client/cifs_unicode.h | 330 +-----------------
fs/smb/client/cifs_uniupr.h | 239 -------------
fs/smb/client/smb2pdu.c | 4 +-
fs/smb/server/Kconfig | 1 +
fs/smb/server/unicode.c | 1 -
fs/smb/server/unicode.h | 325 +----------------
17 files changed, 342 insertions(+), 1166 deletions(-)
delete mode 100644 fs/jfs/jfs_uniupr.c
create mode 100644 fs/nls/nls_ucs2_data.h
rename fs/{smb/server/uniupr.h => nls/nls_ucs2_utils.c} (50%)
create mode 100644 fs/nls/nls_ucs2_utils.h
delete mode 100644 fs/smb/client/cifs_uniupr.hSignature · 1 line
2.41.0
PATCH v5 1/4 fs/smb: Remove unicode 'lower' tables
From: "Dr. David Alan Gilbert" <[email protected]>
The unicode glue in smb/*/..uniupr.h has a section guarded by 'ifndef UNIUPR_NOLOWER' - but that's always defined in smb/*/..unicode.h. Nuke those tables.
--- fs/smb/client/cifs_unicode.h | 50 -------------- fs/smb/client/cifs_uniupr.h | 116 ------------------------------- fs/smb/server/unicode.h | 48 ------------- fs/smb/server/uniupr.h | 129 ----------------------------------- 4 files changed, 343 deletions(-)
fs/smb/client/cifs_unicode.h +0 -50
@@ -22,8 +22,6 @@#include <linux/types.h> #include <linux/nls.h> -#define UNIUPR_NOLOWER /* Example to not expand lower case tables */ -/* * Windows maps these to the user defined 16 bit Unicode range since they are * reserved symbols (along with \ and /), otherwise illegal to store
@@ -84,11 +82,6 @@ extern signed char CifsUniUpperTable[512];extern const struct UniCaseRange CifsUniUpperRange[];#endif /* UNIUPR_NOUPPER */ -#ifndef UNIUPR_NOLOWER -extern signed char CifsUniLowerTable[512 ]; -extern const struct UniCaseRange CifsUniLowerRange[]; -#endif /* UNIUPR_NOLOWER */ -#ifdef __KERNEL__int cifs_from_utf16 (char *to,const __le16 *from,int tolen,int fromlen,const struct nls_table *cp,int map_type);
@@ -358,47 +351,4 @@ UniStrupr(register __le16 *upin) }#endif /* UNIUPR_NOUPPER */ -#ifndef UNIUPR_NOLOWER -/* -* UniTolower: Convert a unicode character to lower case -*/ -static inline wchar_t -UniTolower (registerwchar_t uc) -{ - registerconst struct UniCaseRange *rp; - -if (uc <sizeof (CifsUniLowerTable)) { -/* Latin characters */ -return uc + CifsUniLowerTable[uc];/* Use base tables */ - }else { - rp = CifsUniLowerRange;/* Use range tables */ -while (rp->start) { -if (uc < rp->start)/* Before start of range */ -return uc;/* Uppercase = input */ -if (uc <= rp->end)/* In range */ -return uc + rp->table[uc - rp->start]; - rp++;/* Try next range */ - } - } -return uc;/* Past last range */ -} - -/* -* UniStrlwr: Lower case a unicode string -*/ -static inline wchar_t * -UniStrlwr (registerwchar_t *upin) -{ - registerwchar_t *up; - - up = upin; -while (*up) {/* For all characters */ - *up =UniTolower (*up); - up++; - } -return upin;/* Return input pointer */ -} - -#endif -#endif /* _CIFS_UNICODE_H */
fs/smb/client/cifs_uniupr.h +0 -116
119 lines not on this page · raw message
fs/smb/server/unicode.h +0 -48
64 lines not on this page · raw message
fs/smb/server/uniupr.h +0 -129
133 lines not on this page · raw message
Signature · 1 line
2.41.0
PATCH v5 2/4 fs/smb: Swing unicode common code from smb->NLS
In reply to [email protected]From: "Dr. David Alan Gilbert" <[email protected]>
Swing most of the inline functions and unicode tables into nls from the copy in smb/server. This is UCS-2 rather than most of the rest of the code in NLS, but it currently seems like the best place for it.
The actual unicode.c implementations vary much more between server and client so they're unmoved.
---
fs/nls/Kconfig | 8 +
fs/nls/Makefile | 1 +
.../server/uniupr.h => nls/nls_ucs2_utils.c} | 27 +-
fs/nls/nls_ucs2_utils.h | 297 ++++++++++++++++++
fs/smb/server/Kconfig | 1 +
fs/smb/server/unicode.c | 1 -
fs/smb/server/unicode.h | 277 +---------------
7 files changed, 328 insertions(+), 284 deletions(-)
rename fs/{smb/server/uniupr.h => nls/nls_ucs2_utils.c} (91%)
create mode 100644 fs/nls/nls_ucs2_utils.hfs/nls/Kconfig +8 -0
@@ -617,4 +617,12 @@ config NLS_UTF8 input/output character sets. Say Y here for the UTF-8 encoding of the Unicode/ISO9646 universal character set. +config NLS_UCS2_UTILS + tristate "NLS UCS-2 UTILS" + help + Set of older UCS-2 conversion utilities and tables used by some + filesystems including SMB/CIFS. This includes upper case conversion + tables. This will automatically be selected when the filesystem + that uses it is selected. + endif # NLS
fs/nls/Makefile +1 -0
@@ -54,3 +54,4 @@ obj-$(CONFIG_NLS_MAC_INUIT) += mac-inuit.o obj-$(CONFIG_NLS_MAC_ROMANIAN) += mac-romanian.o obj-$(CONFIG_NLS_MAC_ROMAN) += mac-roman.o obj-$(CONFIG_NLS_MAC_TURKISH) += mac-turkish.o +obj-$(CONFIG_NLS_UCS2_UTILS) += nls_ucs2_utils.o
fs/nls/nls_ucs2_utils.c renamed from fs/smb/server/uniupr.h +17 -10
@@ -1,19 +1,27 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ +// SPDX-License-Identifier: GPL-2.0-or-later /* * Some of the source code in this file came from fs/cifs/uniupr.h * Copyright (c) International Business Machines Corp., 2000,2002 * -* uniupr.h - Unicode compressed case ranges +* Some of the source code in this file came from fs/cifs/cifs_unicode.c +* +* Copyright (c) International Business Machines Corp., 2000,2009 +* Modified by Steve French ([email protected]) +* Modified by Namjae Jeon ([email protected]) * */ -#ifndef __KSMBD_UNIUPR_H -#define __KSMBD_UNIUPR_H +#include <linux/fs.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <asm/unaligned.h> +#include "nls_ucs2_utils.h" + +MODULE_LICENSE ("GPL" ); -#ifndef UNIUPR_NOUPPER /* * Latin upper case */ -signed char SmbUniUpperTable[512 ] = { +signed char NlsUniUpperTable[512 ] = {0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,/* 000-00f */ 0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,/* 010-01f */ 0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,/* 020-02f */
@@ -51,6 +59,7 @@ signed char SmbUniUpperTable[512] = {0 ,-1 ,0 ,-1 ,0 ,-1 ,0 ,-1 ,0 ,-1 ,0 ,-1 ,0 ,-1 ,0 ,-1 ,/* 1e0-1ef */ 0, 0, -1, -2, 0, -1, 0, 0, 0, -1, 0, -1, 0, -1, 0, -1, /* 1f0-1ff */ }; +EXPORT_SYMBOL_GPL (NlsUniUpperTable);/* Upper case range - Greek */ static signed char UniCaseRangeU03a0[47] = {
@@ -126,7 +135,7 @@ static signed char UniCaseRangeUff40[27] = {/* * Upper Case Range */ -const struct UniCaseRange SmbUniUpperRange[] = { +const struct UniCaseRange NlsUniUpperRange[] = { {0x03a0 ,0x03ce , UniCaseRangeU03a0}, {0x0430 ,0x045f , UniCaseRangeU0430}, {0x0490 ,0x04cc , UniCaseRangeU0490},
@@ -134,6 +143,4 @@ const struct UniCaseRange SmbUniUpperRange[] = { {0xff40, 0xff5a, UniCaseRangeUff40}, {0} }; -#endif - -#endif /* __KSMBD_UNIUPR_H */ +EXPORT_SYMBOL_GPL (NlsUniUpperRange);
fs/nls/nls_ucs2_utils.h new file +297 -0
297 lines not on this page · raw message
fs/smb/server/Kconfig +1 -0
@@ -5,6 +5,7 @@ config SMB_SERVER depends on FILE_LOCKING select NLS select NLS_UTF8 + select NLS_UCS2_UTILS select CRYPTO select CRYPTO_MD5 select CRYPTO_HMAC
fs/smb/server/unicode.c +0 -1
@@ -11,7 +11,6 @@#include <asm/unaligned.h> #include "glob.h" #include "unicode.h" -#include "uniupr.h" #include "smb_common.h" /*
fs/smb/server/unicode.h +4 -273
291 lines not on this page · raw message
Signature · 1 line
2.41.0
PATCH v5 3/4 fs/smb/client: Use common code in client
In reply to [email protected]From: "Dr. David Alan Gilbert" <[email protected]>
Now we've got the common code, use it for the client as well. Note there's a change here where we're using the server version of UniStrcat now which had different types (__le16 vs wchar_t) but it's not interpreting the value other than checking for 0, however we do need casts to keep sparse happy.
--- fs/smb/client/Kconfig | 1 + fs/smb/client/cifs_unicode.c | 1 - fs/smb/client/cifs_unicode.h | 280 +---------------------------------- fs/smb/client/cifs_uniupr.h | 123 --------------- fs/smb/client/smb2pdu.c | 4 +- 5 files changed, 4 insertions(+), 405 deletions(-) delete mode 100644 fs/smb/client/cifs_uniupr.h
fs/smb/client/Kconfig +1 -0
@@ -3,6 +3,7 @@ config CIFS tristate "SMB3 and CIFS support (advanced network filesystem)" depends on INET select NLS + select NLS_UCS2_UTILS select CRYPTO select CRYPTO_MD5 select CRYPTO_SHA256
fs/smb/client/cifs_unicode.c +0 -1
@@ -8,7 +8,6 @@#include <linux/slab.h> #include "cifs_fs_sb.h" #include "cifs_unicode.h" -#include "cifs_uniupr.h" #include "cifspdu.h" #include "cifsglob.h" #include "cifs_debug.h"
fs/smb/client/cifs_unicode.h +1 -279
296 lines not on this page · raw message
fs/smb/client/cifs_uniupr.h deleted +0 -123
123 lines not on this page · raw message
fs/smb/client/smb2pdu.c +2 -2
@@ -2570,8 +2570,8 @@ alloc_path_with_tree_prefix(__le16 **out_path, int *out_size, int *out_len,/* Do not append the separator if the path is empty */ if (path[0 ] !=cpu_to_le16 (0x0000 )) { -UniStrcat (*out_path, sep); -UniStrcat (*out_path, path); +UniStrcat ((wchar_t *)*out_path, (wchar_t *)sep); +UniStrcat ((wchar_t *)*out_path, (wchar_t *)path); }unload_nls (cp);
Signature · 1 line
2.41.0
PATCH v5 4/4 fs/jfs: Use common ucs2 upper case table
In reply to [email protected]From: "Dr. David Alan Gilbert" <[email protected]>
Use the UCS-2 upper case tables from nls, that are shared with smb.
This code in JFS is hard to test, so we're only reusing the same tables (which are identical), not trying to reuse the rest of the helper functions.
--- fs/jfs/Kconfig | 1 + fs/jfs/Makefile | 2 +- fs/jfs/jfs_unicode.h | 17 ++---- fs/jfs/jfs_uniupr.c | 121 ---------------------------------------- fs/nls/nls_ucs2_data.h | 15 +++++ fs/nls/nls_ucs2_utils.h | 14 +---- 6 files changed, 23 insertions(+), 147 deletions(-) delete mode 100644 fs/jfs/jfs_uniupr.c create mode 100644 fs/nls/nls_ucs2_data.h
fs/jfs/Kconfig +1 -0
@@ -2,6 +2,7 @@ config JFS_FS tristate "JFS filesystem support" select NLS + select NLS_UCS2_UTILS select CRC32 select LEGACY_DIRECT_IO help
fs/jfs/Makefile +1 -1
@@ -9,7 +9,7 @@ jfs-y := super.o file.o inode.o namei.o jfs_mount.o jfs_umount.o \ jfs_xtree.o jfs_imap.o jfs_debug.o jfs_dmap.o \ jfs_unicode.o jfs_dtree.o jfs_inode.o jfs_discard.o \ jfs_extent.o symlink.o jfs_metapage.o \ - jfs_logmgr.o jfs_txnmgr.o jfs_uniupr.o \ + jfs_logmgr.o jfs_txnmgr.o \ resize.o xattr.o ioctl.o jfs-$(CONFIG_JFS_POSIX_ACL) += acl.o
fs/jfs/jfs_unicode.h +5 -12
@@ -8,16 +8,9 @@#include <linux/slab.h> #include <asm/byteorder.h> +#include "../nls/nls_ucs2_data.h" #include "jfs_types.h" -typedef struct { -wchar_t start; -wchar_t end; -signed char *table; -}UNICASERANGE ; - -extern signed char UniUpperTable[512 ]; -extern UNICASERANGE UniUpperRange[];extern int get_UCSname (struct component_name *,struct dentry *);extern int jfs_strfromUCS_le (char *,const __le16 *,int ,struct nls_table *);
@@ -107,12 +100,12 @@ static inline wchar_t *UniStrncpy_from_le(wchar_t * ucs1, const __le16 * ucs2,*/ static inline wchar_t UniToupper (wchar_t uc) { -UNICASERANGE *rp; +const struct UniCaseRange *rp; -if (uc <sizeof (UniUpperTable)) {/* Latin characters */ -return uc + UniUpperTable[uc];/* Use base tables */ +if (uc <sizeof (NlsUniUpperTable)) {/* Latin characters */ +return uc + NlsUniUpperTable[uc];/* Use base tables */ }else { - rp = UniUpperRange;/* Use range tables */ + rp = NlsUniUpperRange;/* Use range tables */ while (rp->start) {if (uc < rp->start)/* Before start of range */ return uc;/* Uppercase = input */
fs/jfs/jfs_uniupr.c deleted +0 -121
121 lines not on this page · raw message
fs/nls/nls_ucs2_data.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef _NLS_UCS2_DATA_H +#define _NLS_UCS2_DATA_H + +struct UniCaseRange { +wchar_t start; +wchar_t end; +signed char *table; +}; + +extern signed char NlsUniUpperTable[512 ]; +extern const struct UniCaseRange NlsUniUpperRange[]; + +#endif /* _NLS_UCS2_DATA_H */
fs/nls/nls_ucs2_utils.h +1 -13
@@ -26,6 +26,7 @@#include <linux/types.h> #include <linux/nls.h> #include <linux/unicode.h> +#include "nls_ucs2_data.h" /* * Windows maps these to the user defined 16 bit Unicode range since they are
@@ -40,19 +41,6 @@#define UNI_PIPE ((__u16)('|' + 0xF000))#define UNI_SLASH ((__u16)('\\' + 0xF000)) -#ifndef UNICASERANGE_DEFINED -struct UniCaseRange { -wchar_t start; -wchar_t end; -signed char *table; -}; -#endif /* UNICASERANGE_DEFINED */ - -#ifndef UNIUPR_NOUPPER -extern signed char NlsUniUpperTable[512 ]; -extern const struct UniCaseRange NlsUniUpperRange[]; -#endif /* UNIUPR_NOUPPER */ -/* * UniStrcat: Concatenate the second string to the first *
Signature · 1 line
2.41.0
tentatively merged to cifs-2.6.git for-next pending additional testing
Let me know if anyone wants to add RB or Acked-by to any of these or spots any issues
Quoted 64 lines · On Wed, Aug 16, 2023 at 7:22 PM <[email protected]> wrote:
From: "Dr. David Alan Gilbert" <[email protected]>
The smb client and server code have (mostly) duplicated code for unicode manipulation, in particular upper case handling.
Flatten this lot into shared code.
There's some code that's slightly different between the two, and I've not attempted to share that - this should be strictly a no behaviour change set.
In addition, the same tables and code are shared in jfs, however there's very little testing available for the unicode in there, so just share the raw data tables.
I suspect there's more UCS-2 code that can be shared, in the NLS code and in the UCS-2 code used by the EFI interfaces.
Lightly tested with a module and a monolithic build, and just mounting itself.
This dupe was found using PMD:
https://pmd.github.io/pmd/pmd_userdocs_cpd.htmlDave
Version 5
Add some (wchar_t*) casts to keep sparse
happy, as spotted by kernel test robotDr. David Alan Gilbert (4):
fs/smb: Remove unicode 'lower' tables
fs/smb: Swing unicode common code fromsmb->NLS
fs/smb/client: Use common code in client
fs/jfs: Use common ucs2 upper case tablefs/jfs/Kconfig | 1 + fs/jfs/Makefile | 2 +- fs/jfs/jfs_unicode.h | 17 +- fs/jfs/jfs_uniupr.c | 121 ------- fs/nls/Kconfig | 8 + fs/nls/Makefile | 1 + fs/nls/nls_ucs2_data.h | 15 + .../server/uniupr.h => nls/nls_ucs2_utils.c} | 156 +-------- fs/nls/nls_ucs2_utils.h | 285 +++++++++++++++ fs/smb/client/Kconfig | 1 + fs/smb/client/cifs_unicode.c | 1 - fs/smb/client/cifs_unicode.h | 330 +----------------- fs/smb/client/cifs_uniupr.h | 239 ------------- fs/smb/client/smb2pdu.c | 4 +- fs/smb/server/Kconfig | 1 + fs/smb/server/unicode.c | 1 - fs/smb/server/unicode.h | 325 +---------------- 17 files changed, 342 insertions(+), 1166 deletions(-) delete mode 100644 fs/jfs/jfs_uniupr.c create mode 100644 fs/nls/nls_ucs2_data.h rename fs/{smb/server/uniupr.h => nls/nls_ucs2_utils.c} (50%) create mode 100644 fs/nls/nls_ucs2_utils.h delete mode 100644 fs/smb/client/cifs_uniupr.h--
2.41.0
Signature · 3 lines
Thanks, Steve