isofs: bound the name conversion and return long Joliet names whole
Subsystems (2)
v2 cover isofs: bound the name conversion and return long Joliet names whole
Honza, this reworks v1 along the lines of your review, on top of yourfor_next: both converters take the buffer size, the callers pass it, and the buffer shrinks to what they need, without size asserts. But on long Joliet names it deliberately goes the other way: 1/4 returns them whole instead of cutting them at NAME_MAX, and 2/4 reports the longer limit in statfs. 1/4 also corrects the v1 history of the PAGE_SIZE bound.
1/4 is also a fix. get_joliet_filename() keeps the converted length in an unsigned char, so a Joliet name longer than 255 bytes of UTF-8 wraps and is listed under a garbled short name that cannot be looked up. PowerISO and UltraISO write such names by default (up to 110 UTF-16 units, 330 bytes for CJK).
Such names are rare: in the Joliet trees of a few thousand archive.org images, mostly CJK, Thai or Indic, none is longer than 255 bytes. The images I wrote with PowerISO and UltraISO (under wine), a crafted image with 111-unit names, and the survey scripts are at
https://github.com/matthiasgoergens/linux/tree/isofs-joliet-long-names
(or I can post them here).
Why whole: as with empty directory blocks [1], I'd follow Windows. Joliet is Microsoft's extension, discs with such names are made with Windows tools for people who read them on Windows, and Windows returns the names whole, including 111-unit names in records that leave out the padding byte. Scripts and CI logs:
https://github.com/matthiasgoergens/isofs-windows-probe
On Linux, vfat and exfat already return names of up to 765 bytes and report a limit of 1530 in statfs, vfat since f68e542f3478 ("fat: Fix statfs->f_namelen"), and the VFS itself only limits a name to PATH_MAX (verify_dirent_name()). So what breaks is what breaks with vfat today:
copying such a file to ext4 or tmpfs fails with ENAMETOOLONG, which cp, tar and rsync report before carrying on; readdir_r() and inotify readers sized by the man page fail; and fanotify reports the event without the name and hits the WARN_ON_ONCE() in fanotify_info_copy_name(), for which I have sent a fix [2]. 1/4 has the full list. Cutting at NAME_MAX avoids all of this, but lists names that Windows does not show and that can collide within a directory.
If you still prefer NAME_MAX, I have that version ready and tested the same way: there 1/4 cuts long names at NAME_MAX on a character boundary, 2/4 is dropped, and 4/4 shrinks the buffer to NAME_MAX + 1.
Testing: a KASAN and UBSAN kernel under qemu, 19 images mounted with no iocharset, utf8, iso8859-1, cp932 and euc-jp, each with and without norock. Only the four images with names over 255 bytes change, and only with no iocharset or utf8: their long names are now listed whole and open. Everything else lists, looks up and reads as before. Under a Debian userspace, ls, find, cp, tar, rsync, Python, readdir_r(), inotify and fanotify behave on the long-name images, including a crafted one with 111-unit names, as they do on a vfat with 765-byte names. This replaces v1 2/2's claim, whose test never ran the Joliet or zisofs code.
v1: https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/cmlro2xzle2aa7ebflvxhbcv74mea7p6qvxhin6egpdvyzyuxh@s45tpgxdal3n/
[2] https://lore.kernel.org/all/[email protected]/
Matthias Goergens (4):
isofs: return long Joliet names whole
isofs: report the Joliet name length in statfs
isofs: pass the name buffer size to get_rock_ridge_filename()
isofs: shrink the name conversion buffer
fs/isofs/dir.c | 9 ++++++--- fs/isofs/inode.c | 3 ++- fs/isofs/isofs.h | 14 ++++++++++++-- fs/isofs/joliet.c | 27 ++++++++++++++++++++------- fs/isofs/namei.c | 8 +++++--- fs/isofs/rock.c | 8 ++++++-- 6 files changed, 51 insertions(+), 18 deletions(-)
base-commit: c8437ca3d4386af1ae1f2869643e82fdb9c1f0f5
Signature · 1 line
2.55.0
PATCH v2 1/4 isofs: return long Joliet names whole
get_joliet_filename() writes into a buffer allocated by its caller but is not told how big that buffer is. It hardcodes PAGE_SIZE as the output limit for utf16s_to_utf8s() and gives uni16_to_x8() no limit at all. PAGE_SIZE was never the right bound. Before commit b2eb2e288604 ("isofs: Drop support of directory entries straddling blocks") both callers allocated a page, but only its first 1024 bytes were for the name, the rest holding a copy of a directory record that straddled a block. It could not overflow only because a directory record holds at most 222 bytes of name (255 bytes, 33 of them fixed), which no converter expands beyond 1024 bytes.
It also keeps the converted length in an unsigned char. 222 bytes of name are 111 UTF-16 units, and the UTF-8 converter, used for iocharset=utf8 and when CONFIG_NLS_DEFAULT is "utf8", needs three bytes for each CJK, Thai or Devanagari character, so such a name can take up to 333 bytes. Past 255 the length wraps. readdir then reports a short name that usually ends in the middle of a UTF-8 sequence, and lookup finds the file only under that name, never under its real one; a name that wraps to exactly 0 bytes is not listed at all.
Such names are out of spec, since Joliet allows 64 units, but common tools write them: PowerISO 9.5 and UltraISO 9.76 both keep up to 110 units by default. Windows, for which Joliet was made, returns them whole, including 111-unit names in records that leave out the padding byte.
Return them whole here too. Pass the buffer size down and have both converters respect it. utf16s_to_utf8s() stops before a character that does not fit, and uni16_to_x8() now hands uni2char() the space that actually remains and stops on -ENAMETOOLONG, as fs/hfsplus/unicode.c does. Both callers pass JOLIET_NAME_MAX + 1, room for 111 units at three bytes each plus the terminator; no character set needs more than three bytes for a UTF-16 unit, and isofs_dir_record_valid() already rejects a record whose name_len claims more than the record holds, so no name is cut. Make the length an int, which is what both converters return.
These names are longer than NAME_MAX. POSIX lets the limit vary by filesystem, reported by pathconf(_PC_NAME_MAX), and the VFS limits a name only by PATH_MAX (verify_dirent_name() in fs/readdir.c).
vfat, exfat, hfsplus and ntfs3 already return names of up to 255 UTF-16 units, 765 bytes of UTF-8, and such names break the same things there:
copying the file to a filesystem with a 255-byte limit, such as ext4 or tmpfs, fails with ENAMETOOLONG, and cp, tar, rsync and Python's shutil report that file and carry on with the rest, so the copy is incomplete but says so. glibc's readdir() returns the names, but the deprecated readdir_r() skips them and fails with ENAMETOOLONG, and an inotify reader with the buffer size inotify(7) suggests gets EINVAL. In the kernel, fanotify reports events on such a file without its name and warns once in fanotify_info_copy_name(), and a directory with such a name cannot be reconnected from a file handle, because the generic get_name() in fs/exportfs, which isofs uses, skips names longer than NAME_MAX. Cutting at NAME_MAX would avoid these, but would list names that Windows does not show and that can coincide within a directory.
Every name of at most 255 bytes is returned exactly as before.
--- fs/isofs/dir.c | 4 +++- fs/isofs/isofs.h | 11 ++++++++++- fs/isofs/joliet.c | 27 ++++++++++++++++++++------- fs/isofs/namei.c | 3 ++- 4 files changed, 35 insertions(+), 10 deletions(-)
fs/isofs/dir.c +3 -1
@@ -195,7 +195,9 @@ static int do_isofs_readdir(struct inode *inode, struct file *file,if (map) {#ifdef CONFIG_JOLIET if (sbi->s_joliet_level) { - len =get_joliet_filename (de, tmpname, inode); + len =get_joliet_filename (de, tmpname, +JOLIET_NAME_MAX +1 , + inode); p = tmpname; } else#endif
fs/isofs/isofs.h +10 -1
@@ -121,7 +121,16 @@ bool isofs_dir_record_valid(struct iso_directory_record *de,unsigned long offset, unsigned long bufsize); -int get_joliet_filename (struct iso_directory_record *,unsigned char *,struct inode *); +/* +* The longest name the Joliet converter returns, in bytes. A directory +* record is at most 255 bytes long, which leaves room for 111 UTF-16 units +* of name, and no character set needs more than three bytes for one unit. +*/ +#define JOLIET_NAME_MAX \ + ((255 - sizeof(struct iso_directory_record)) / 2 * 3) + +int get_joliet_filename (struct iso_directory_record *de,unsigned char *outname, +int outsize,struct inode *inode);int get_acorn_filename (struct iso_directory_record *,char *,struct inode *);extern struct dentry *isofs_lookup (struct inode *,struct dentry *,unsigned int flags);
fs/isofs/joliet.c +20 -7
@@ -15,19 +15,26 @@* Convert Unicode 16 to UTF-8 or ASCII. */ static int -uni16_to_x8 (unsigned char *ascii,__be16 *uni,int len,struct nls_table *nls) +uni16_to_x8 (unsigned char *ascii,__be16 *uni,int len,struct nls_table *nls, +int outsize) {__be16 *ip, ch; -unsigned char *op; +unsigned char *op, *end; ip = uni; op = ascii; + end = ascii + outsize -1 ;/* leave room for the terminator */ while ((ch =get_unaligned (ip)) && len) {int llen; - llen = nls->uni2char (be16_to_cpu (ch), op,NLS_MAX_CHARSET_SIZE ); + +if (op >= end) +break ; + llen = nls->uni2char (be16_to_cpu (ch), op, end - op);if (llen >0 ) op += llen; +else if (llen == -ENAMETOOLONG ) +break ;else *op++ ='?' ; ip++;
@@ -38,21 +45,27 @@ uni16_to_x8(unsigned char *ascii, __be16 *uni, int len, struct nls_table *nls)return (op - ascii); } +/* +* Convert the Joliet name of @de into @outname, a buffer of @outsize bytes. +* The result is at most @outsize - 1 bytes long; a longer name is cut at a +* character boundary. +*/ int -get_joliet_filename (struct iso_directory_record * de,unsigned char *outname,struct inode * inode) +get_joliet_filename (struct iso_directory_record *de,unsigned char *outname, +int outsize,struct inode *inode) {struct nls_table *nls; -unsigned char len =0 ; +int len =0 ; nls =ISOFS_SB (inode->i_sb)->s_nls_iocharset;if (!nls) { len =utf16s_to_utf8s ((const wchar_t *) de->name, de->name_len[0 ] >>1 ,UTF16_BIG_ENDIAN , - outname,PAGE_SIZE ); + outname, outsize -1 ); }else { len =uni16_to_x8 (outname, (__be16 *) de->name, - de->name_len[0 ] >>1 , nls); + de->name_len[0 ] >>1 , nls, outsize); }if ((len >2 ) && (outname[len-2 ] ==';' ) && (outname[len-1 ] =='1' )) len -=2 ;
fs/isofs/namei.c +2 -1
@@ -109,7 +109,8 @@ isofs_find_entry(struct inode *dir, struct dentry *dentry, dpnt = tmpname; #ifdef CONFIG_JOLIET }else if (sbi->s_joliet_level) { - dlen =get_joliet_filename (de, tmpname, dir); + dlen =get_joliet_filename (de, tmpname, +JOLIET_NAME_MAX +1 , dir); dpnt = tmpname;#endif }else if (sbi->s_mapping =='a' ) {
Signature · 1 line
2.55.0
PATCH v2 2/4 isofs: report the Joliet name length in statfs
In reply to Matthias Goergensisofs_statfs() reports NAME_MAX as the longest name, but a Joliet name can now be up to JOLIET_NAME_MAX bytes long. glibc answers pathconf(_PC_NAME_MAX) from f_namelen, so a program sizing a buffer from it should get room for every name readdir returns. vfat has reported 255 times NLS_MAX_CHARSET_SIZE, 1530, since commit f68e542f3478 ("fat:
Fix statfs->f_namelen"), which followed a report of readdir_r() overflowing a buffer sized that way, and exfat reports the same. Windows reports a maximum component length of 110 for CDFS.
Report JOLIET_NAME_MAX on Joliet mounts. Rock Ridge and plain ISO 9660 mounts keep NAME_MAX.
--- fs/isofs/inode.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/isofs/inode.c +2 -1
@@ -1012,7 +1012,8 @@ static int isofs_statfs (struct dentry *dentry, struct kstatfs *buf) buf->f_files =ISOFS_SB (sb)->s_ninodes; buf->f_ffree =0 ; buf->f_fsid =u64_to_fsid (id); - buf->f_namelen =NAME_MAX ; + buf->f_namelen =ISOFS_SB (sb)->s_joliet_level ? +JOLIET_NAME_MAX :NAME_MAX ;return 0 ; }
Signature · 1 line
2.55.0
PATCH v2 3/4 isofs: pass the name buffer size to get_rock_ridge_filename()
In reply to Matthias Goergensget_rock_ridge_filename() assembles a name from one or more NM entries into a buffer allocated by its caller. It is not told how big that buffer is, and instead stops adding NM entries once the name would exceed NAME_MAX, relying on the callers' buffer being larger.
Pass the size in and check against it. Both callers pass NAME_MAX + 1, so the behaviour is unchanged: an NM entry that would take the name past 255 bytes is dropped, together with any that follow, and the entries before it are returned.
--- fs/isofs/dir.c | 3 ++- fs/isofs/isofs.h | 3 ++- fs/isofs/namei.c | 3 ++- fs/isofs/rock.c | 8 ++++++-- 4 files changed, 12 insertions(+), 5 deletions(-)
fs/isofs/dir.c +2 -1
@@ -186,7 +186,8 @@ static int do_isofs_readdir(struct inode *inode, struct file *file, map =1 ;if (sbi->s_rock) { - len =get_rock_ridge_filename (de, tmpname, inode); + len =get_rock_ridge_filename (de, tmpname,NAME_MAX +1 , + inode);if (len !=0 ) {/* may be -1 */ p = tmpname; map =0 ;
fs/isofs/isofs.h +2 -1
@@ -115,7 +115,8 @@ struct timespec64 iso_date(u8 *p, int flags);struct inode ;/* To make gcc happy */ extern int parse_rock_ridge_inode (struct iso_directory_record *,struct inode *,int relocated); -extern int get_rock_ridge_filename (struct iso_directory_record *,char *,struct inode *); +int get_rock_ridge_filename (struct iso_directory_record *de,char *retname, +int retnamesize,struct inode *inode);extern int isofs_name_translate (struct iso_directory_record *,char *,struct inode *); bool isofs_dir_record_valid(struct iso_directory_record *de, unsigned long offset,
fs/isofs/namei.c +2 -1
@@ -104,7 +104,8 @@ isofs_find_entry(struct inode *dir, struct dentry *dentry, dpnt = de->name;if (sbi->s_rock && - ((i =get_rock_ridge_filename (de, tmpname, dir)))) { + ((i =get_rock_ridge_filename (de, tmpname,NAME_MAX +1 , + dir)))) { dlen = i;/* possibly -1 */ dpnt = tmpname;#ifdef CONFIG_JOLIET
fs/isofs/rock.c +6 -2
@@ -209,10 +209,14 @@ static int rock_check_overflow(struct rock_state *rs, int sig) }/* +* Build the Rock Ridge name of @de in @retname, a buffer of @retnamesize +* bytes. From the first NM entry that does not fit along with the +* terminator, the rest of the name is dropped. +* * return length of name field; 0: not found, -1: to be ignored */ int get_rock_ridge_filename (struct iso_directory_record *de, -char *retname,struct inode *inode) +char *retname,int retnamesize,struct inode *inode) {struct rock_state rs;struct rock_ridge *rr;
@@ -287,7 +291,7 @@ int get_rock_ridge_filename(struct iso_directory_record *de,break ; } len = rr->len -5 ; -if (retnamlen + len >NAME_MAX ) { +if (retnamlen + len >= retnamesize) { truncate =1 ;break ; }
Signature · 1 line
2.55.0
PATCH v2 4/4 isofs: shrink the name conversion buffer
In reply to Matthias Goergensisofs_readdir() and isofs_lookup() allocate 1024 bytes for the name converters. Now that get_rock_ridge_filename() and get_joliet_filename() are told the buffer size, nothing writes beyond JOLIET_NAME_MAX + 1 bytes: get_joliet_filename() is passed that size, get_rock_ridge_filename() is passed NAME_MAX + 1, isofs_name_translate() copies at most the 222 bytes of name a directory record can hold, and get_acorn_filename() appends at most five bytes to that.
Allocate JOLIET_NAME_MAX + 1 bytes, matching what the Joliet callers pass.
No functional change.
--- fs/isofs/dir.c | 2 +- fs/isofs/namei.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
fs/isofs/dir.c +1 -1
@@ -236,7 +236,7 @@ static int isofs_readdir(struct file *file, struct dir_context *ctx)char *tmpname;struct inode *inode =file_inode (file); - tmpname =kmalloc (1024 ,GFP_KERNEL ); + tmpname =kmalloc (JOLIET_NAME_MAX +1 ,GFP_KERNEL );if (tmpname ==NULL )return -ENOMEM ;
fs/isofs/namei.c +1 -1
@@ -157,7 +157,7 @@ struct dentry *isofs_lookup(struct inode *dir, struct dentry *dentry, unsigned istruct inode *inode;char *tmpname; - tmpname =kmalloc (1024 ,GFP_USER ); + tmpname =kmalloc (JOLIET_NAME_MAX +1 ,GFP_USER );if (!tmpname)return ERR_PTR (-ENOMEM );
Signature · 1 line
2.55.0
Jan Kara
maintainerQuoted 56 lines · Matthias Goergens
Honza, this reworks v1 along the lines of your review, on top of your
for_next: both converters take the buffer size, the callers pass it, and the buffer shrinks to what they need, without size asserts. But on long Joliet names it deliberately goes the other way: 1/4 returns them whole instead of cutting them atNAME_MAX, and 2/4 reports the longer limit in statfs. 1/4 also corrects the v1 history of thePAGE_SIZEbound.1/4 is also a fix.
get_joliet_filename()keeps the converted length in an unsigned char, so a Joliet name longer than 255 bytes of UTF-8 wraps and is listed under a garbled short name that cannot be looked up. PowerISO and UltraISO write such names by default (up to 110 UTF-16 units, 330 bytes for CJK).Such names are rare: in the Joliet trees of a few thousand archive.org images, mostly CJK, Thai or Indic, none is longer than 255 bytes. The images I wrote with PowerISO and UltraISO (under wine), a crafted image with 111-unit names, and the survey scripts are at
https://github.com/matthiasgoergens/linux/tree/isofs-joliet-long-names
(or I can post them here).
Why whole: as with empty directory blocks [1], I'd follow Windows. Joliet is Microsoft's extension, discs with such names are made with Windows tools for people who read them on Windows, and Windows returns the names whole, including 111-unit names in records that leave out the padding byte. Scripts and CI logs:
https://github.com/matthiasgoergens/isofs-windows-probe
On Linux, vfat and exfat already return names of up to 765 bytes and report a limit of 1530 in statfs, vfat since
f68e542f3478("fat: Fixstatfs->f_namelen"), and the VFS itself only limits a name toPATH_MAX(verify_dirent_name()). So what breaks is what breaks with vfat today:
copying such a file to ext4 or tmpfs fails with ENAMETOOLONG, which cp, tar and rsync report before carrying on;readdir_r()and inotify readers sized by the man page fail; and fanotify reports the event without the name and hits theWARN_ON_ONCE()infanotify_info_copy_name(), for which I have sent a fix [2]. 1/4 has the full list. Cutting atNAME_MAXavoids all of this, but lists names that Windows does not show and that can collide within a directory.If you still prefer
NAME_MAX, I have that version ready and tested the same way: there 1/4 cuts long names atNAME_MAXon a character boundary, 2/4 is dropped, and 4/4 shrinks the buffer toNAME_MAX+ 1.Testing: a KASAN and UBSAN kernel under qemu, 19 images mounted with no iocharset, utf8, iso8859-1, cp932 and euc-jp, each with and without norock. Only the four images with names over 255 bytes change, and only with no iocharset or utf8: their long names are now listed whole and open. Everything else lists, looks up and reads as before. Under a Debian userspace, ls, find, cp, tar, rsync, Python,
readdir_r(), inotify and fanotify behave on the long-name images, including a crafted one with 111-unit names, as they do on a vfat with 765-byte names. This replaces v1 2/2's claim, whose test never ran the Joliet or zisofs code.
Quoted 3 lines
v1: https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/cmlro2xzle2aa7ebflvxhbcv74mea7p6qvxhin6egpdvyzyuxh@s45tpgxdal3n/
[2] https://lore.kernel.org/all/[email protected]/
Thanks! The patches look good and I've added them to my tree. I've just added the following hunk to your last patch to document the situation in the code:
+ /*
+ * Rockridge can produce names of NAME_MAX size, Acorn extensions at + * most 227 chars. + */
+ BUILD_BUG_ON(JOLIET_NAME_MAX < NAME_MAX || JOLIET_NAME_MAX < 227);
I've also noticed we are somewhat inconsistent with the terminating \0 character. In particular get_joliet_filename() -> utf16s_to_utf8s() doesn't null-terminate the returned string. Similarly isofs_name_translate() doesn't result in null-terminated string. Other functions do terminate it (which isn't really necessary AFAICS). It would be nice to clean this up if you're interested.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR