fat problem in 2.4.2
Subsystems (4)
The bug with truncate in the fat filesystem that was present in 2.4.0, and fixed with the 2.4.0-ac12 (or earlier) patch is still in the main (unpatched) kernel, both 2.4.1 and 2.4.2. The problem is that I cannot apply the 2.4.0-ac patch to the newer kernels and I cannot patch up to 2.4.1 from 2.4.0 if I already applied the ac patch (so I obviously can't get to 2.4.2). I also can't patch from 2.4.0-ac12 to 2.4.1-ac20 using only ac patches. Neither of the ac patches for 2.4.1 and 2.4.2 have this fix for fat, so giving the '-R' option to patch took away the fix.
I would be happy to extract the fix from 2.4.0-ac12, but I am not a kernel developer and am not sure how many files were changed to fix this bug.
I want to move up to 2.4.2 for the reiserfs updates and then eventually to at least 2.4.2-ac2 for the loopback fix (but need the fat functionality too).
Thanks for any help or suggestions.
James Strandboge
[email protected]
Quoted 2 lines
The bug with truncate in the fat filesystem that was present in 2.4.0, and fixed with the 2.4.0-ac12 (or earlier) patch is still in the main
It isnt a bug. The fix in 2.4-ac I've dropped. A program that assumes ftruncating a file large will work is broken.
Alan
Quoted 5 lines · Alan Cox
Quoted 2 lines
The bug with truncate in the fat filesystem that was present in 2.4.0, and fixed with the 2.4.0-ac12 (or earlier) patch is still in the main
It isnt a bug. The fix in 2.4-ac I've dropped. A program that assumes ftruncating a file large will work is broken.
In that case, why was it changed for FAT only? Ext2 will still happily enlarge a file by truncating it.
If the behavior has to be changed, wouldn't it be better to first give people a chance to get programs, that rely on the old behavior fixed, before enforcing the change?
Staroffice (the binary-only version; the new "open source" version is not yet ready for real-world use) for example currently doesn't write to FAT filesystems anymore - which is pretty annoying for people who need it.
Is there somewhere a patch for the current kernel?
Regards,
Peter Daum
Quoted 2 lines · Peter Daum
In that case, why was it changed for FAT only? Ext2 will still happily enlarge a file by truncating it.
Basically, the program depends on behaviour that was never guaranteed to be there.
Quoted 4 lines
Staroffice (the binary-only version; the new "open source" version is not yet ready for real-world use) for example currently doesn't write to FAT filesystems anymore - which is pretty annoying for people who need it.
Staroffice is non-portable and badly written, film at 11...
BTW, _some_ subset is doable on FAT. You can't always do it (bloody thing doesn't support holes), but you can try the following (warning - untested patch):
fs/fat/inode.c +39 -3
@@ -897,6 +897,36 @@unlock_kernel (); } +int generic_cont_expand (struct inode *inode,loff_t size) +{ +struct address_space *mapping = inode->i_mapping; +struct page *page; +unsigned long index, offset, limit; +int err; + + limit = current->rlim[RLIMIT_FSIZE ].rlim_cur; +if (limit !=RLIM_INFINITY ) { +if (size > limit) { +send_sig (SIGXFSZ , current,0 ); + size = limit; + } + } + offset = (size & (PAGE_CACHE_SIZE -1 ));/* Within page */ + index = size >>PAGE_CACHE_SHIFT ; + err = -ENOMEM ; + page =grab_cache_page (mapping, index); +if (!page) + goto out; + err = mapping->a_ops->prepare_write (NULL , page, offset, offset); +if (!err) + err = mapping->a_ops->commit_write (NULL , page, offset, offset); +UnlockPage (page); +page_cache_release (page); +if (err >0 ) + err =0 ; +out: +return err; +}int fat_notify_change (struct dentry * dentry,struct iattr * attr) {
@@ -904,11 +934,17 @@struct inode *inode = dentry->d_inode; -int error; +int error =0 ; -/* FAT cannot truncate to a longer file */ +/* +* On FAT truncate to a longer file may fail with -ENOSPC. No +* way to report it from fat_truncate(), so... +*/ if (attr->ia_valid &ATTR_SIZE ) {if (attr->ia_size > inode->i_size) -return -EPERM ; + error =generic_cont_expand (inode, attr->ia_size); } + +if (error) +return error; error =inode_change_ok (inode, attr);if (error)
That said, if your only problem is Staroffice... <shrug> I would rather rm the crapware and forget about it, but YMMV.
Cheers,
Al
Alexander Viro writes:
[about file expansion by truncate]
Quoted 2 lines
Basically, the program depends on behaviour that was never guaranteed to be there.
1. it is useful
2. it is documented in a few places AFAIK
3. it is portable enough for Star Office (Solaris I guess)
Quoted 3 lines
BTW, _some_ subset is doable on FAT. You can't always do it (bloody thing doesn't support holes), but you can try the following (warning - untested patch):
Holes are nothing special. They are just a simple type of compression. We are doing overcommit on filesystems, and need an OOD killer to wipe out big files like the Star Office executable.
Quoted 2 lines
In that case, why was it changed for FAT only? Ext2 will still happily enlarge a file by truncating it.
ftruncate() and truncate() may extend a file but they are not required to do so.
Quoted 3 lines
If the behavior has to be changed, wouldn't it be better to first give people a chance to get programs, that rely on the old behavior fixed, before enforcing the change?
A program relying on the old behaviour was violating standards. Also its been this way for almost two years.
Quoted 6 lines
Staroffice (the binary-only version; the new "open source" version is not yet ready for real-world use) for example currently doesn't write to FAT filesystems anymore - which is pretty annoying for people who need it.
Is there somewhere a patch for the current kernel?
You might be able to fish it out of old -ac kernel trees and debug it further. Alternatively you could implement it in glibc of course, which is a nicer solution
Quoted 18 lines · Alan Cox
Quoted 2 lines
In that case, why was it changed for FAT only? Ext2 will still happily enlarge a file by truncating it.
ftruncate()andtruncate()may extend a file but they are not required to do so.Quoted 3 lines
If the behavior has to be changed, wouldn't it be better to first give people a chance to get programs, that rely on the old behavior fixed, before enforcing the change?
A program relying on the old behaviour was violating standards. Also its been this way for almost two years.
Quoted 5 lines
Staroffice (the binary-only version; the new "open source" version is not yet ready for real-world use) for example currently doesn't write to FAT filesystems anymore - which is pretty annoying for people who need it.
Quoted 5 lines
Quoted 1 line
Is there somewhere a patch for the current kernel?
You might be able to fish it out of old -ac kernel trees and debug it further. Alternatively you could implement it in glibc of course, which is a nicer solution
Alan, fix is really quite simple. Especially if you have vmtruncate() returning int (ac1 used to do it, I didn't check later ones). Actually just a generic_cont_expand() done on expanding path in vmtruncate() will be enough - it should be OK for all cases, including normal filesystems. <grabbing -ac7>
OK, any brave soul to test that? All I can promise that it builds.
Cheers,
Al
fs/affs/file.c +0 -19
@@ -663,25 +663,6 @@ net_blocksize = blocksize - ((inode->i_sb->u.affs_sb.s_flags &SF_OFS ) ?24 :0 ); first = inode->i_size + net_blocksize -1 ;do_div (first, net_blocksize); -if (inode->u.affs_i.i_lastblock < first -1 ) { -/* There has to be at least one new block to be allocated */ -if (!inode->u.affs_i.i_ec &&alloc_ext_cache (inode)) { -/* XXX Fine! No way to indicate an error. */ -return /* -ENOSPC */ ; - } - bh =affs_getblock (inode,first -1 ); -if (!bh) { -affs_warning (inode->i_sb,"truncate" ,"Cannot extend file" ); - inode->i_size = net_blocksize * (inode->u.affs_i.i_lastblock +1 ); - }else if (inode->i_sb->u.affs_sb.s_flags &SF_OFS ) { - tmp = inode->i_size; - rem =do_div (tmp, net_blocksize); -DATA_FRONT (bh)->data_size =cpu_to_be32 (rem ? rem : net_blocksize); -affs_fix_checksum (blocksize,bh->b_data,5 ); -mark_buffer_dirty (bh); - } - goto out_truncate; - } ekey = inode->i_ino; ext =0 ;
fs/fat/inode.c +0 -6
@@ -904,12 +904,6 @@struct inode *inode = dentry->d_inode;int error; -/* FAT cannot truncate to a longer file */ -if (attr->ia_valid &ATTR_SIZE ) { -if (attr->ia_size > inode->i_size) -return -EPERM ; - } - error =inode_change_ok (inode, attr);if (error)return MSDOS_SB (sb)->options.quiet ?0 : error;
mm/memory.c +34 -4
@@ -924,7 +924,32 @@zap_page_range (mm, start, len); }while ((mpnt = mpnt->vm_next_share) !=NULL ); } - + +static int generic_vm_expand (struct address_space *mapping,loff_t size) +{ +struct page *page; +unsigned long index, offset; +int err; + +if (!mapping->a_ops->prepare_write || !mapping->a_ops->commit_write) +return -ENOSYS ; + + offset = (size & (PAGE_CACHE_SIZE -1 ));/* Within page */ + index = size >>PAGE_CACHE_SHIFT ; + err = -ENOMEM ; + page =grab_cache_page (mapping, index); +if (!page) + goto out; + err = mapping->a_ops->prepare_write (NULL , page, offset, offset); +if (!err) + err = mapping->a_ops->commit_write (NULL , page, offset, offset); +UnlockPage (page); +page_cache_release (page); +if (err >0 ) + err =0 ; +out: +return err; +}/* * Handle all mappings that got truncated by a "truncate()"
@@ -939,6 +964,7 @@unsigned long partial, pgoff;struct address_space *mapping = inode->i_mapping;unsigned long limit; +int err;if (inode->i_size < offset) goto do_expand;
@@ -976,10 +1002,14 @@ offset = limit; } } - inode->i_size = offset; -if (inode->i_op && inode->i_op->truncate) + err =generic_vm_expand (mapping, offset); +if (err == -ENOSYS ) { + err =0 ; + inode->i_size = offset; + } +if (!err && inode->i_op && inode->i_op->truncate) inode->i_op->truncate (inode); -return 0 ; +return err; out:return -EFBIG ; }
Hi,
mm/memory.c line 938
Quoted 11 lines · Alexander Viro
mm/memory.c +11
-0+static int generic_vm_expand (struct address_space *mapping,loff_t size) +{ +struct page *page; +unsigned long index, offset; +int err; + +if (!mapping->a_ops->prepare_write || !mapping->a_ops->commit_write) +return -ENOSYS ; + + offset = (size & (PAGE_CACHE_SIZE -1 ));/* Within page */ + index = size >>PAGE_CACHE_SHIFT ;
For affs I did basically the same with a small difference:
offset = ((size-1) & (PAGE_CACHE_SIZE-1)) + 1; index = (size-1) >> PAGE_CACHE_SHIFT;
That works fine here and allocates a page in the cache more likely to be used.
bye, Roman
Quoted 18 lines · Roman Zippel
Hi,
mm/memory.c line 938
Quoted 11 lines · Alexander Viro
mm/memory.c +11
-0+static int generic_vm_expand (struct address_space *mapping,loff_t size) +{ +struct page *page; +unsigned long index, offset; +int err; + +if (!mapping->a_ops->prepare_write || !mapping->a_ops->commit_write) +return -ENOSYS ; + + offset = (size & (PAGE_CACHE_SIZE -1 ));/* Within page */ + index = size >>PAGE_CACHE_SHIFT ;For affs I did basically the same with a small difference:
Quoted 5 lines
offset = ((size-1) & (PAGE_CACHE_SIZE-1)) + 1; index = (size-1) >> PAGE_CACHE_SHIFT;That works fine here and allocates a page in the cache more likely to be used.
The only difference is in the case when size is a multiple ofPAGE_CACHE_SHIFT, so most of the time it's the same, but yeah, this variant is probably nicer.
The problem with putting it into ->truncate() is obvious -
handling errors. And doing the thing before vmtruncate() (in foo_notify_change(), whatever) is also PITA - you need to reproduce the rlimit handling. Not nice...
IOW, if it's worth doing at all it probably should be
on expanding path in vmtruncate() - limit checks are already done, but old i_size is still not lost...
Cheers,
Al
Hi,
Quoted 3 lines · Alexander Viro
IOW, if it's worth doing at all it probably should be
on expanding path invmtruncate()- limit checks are already done, but oldi_sizeis still not lost...
The fs where it's important have mmu_private, that's what I use to decide whether to expand or truncate.
bye, Roman
In article <[email protected]>,
Quoted 8 lines · Alexander Viro
Alan, fix is really quite simple. Especially if you have
vmtruncate()returning int (ac1 used to do it, I didn't check later ones). Actually just ageneric_cont_expand()done on expanding path invmtruncate()will be enough - it should be OK for all cases, including normal filesystems. <grabbing -ac7>OK, any brave soul to test that? All I can promise that it builds.
This looks like it would create a dummy block even for non-broken filesystems (ie truncating a file to be larger on ext2 would create a block, no?). While that would work, it would also waste disk-space.
Linus
Quoted 9 lines · On Thursday, March 01, 2001 12:05:50 PM -0800 Linus Torvalds…
In article <[email protected]>,
Quoted 7 lines · Alexander Viro
Alan, fix is really quite simple. Especially if you have
vmtruncate()returning int (ac1 used to do it, I didn't check later ones). Actually just ageneric_cont_expand()done on expanding path invmtruncate()will be enough - it should be OK for all cases, including normal filesystems. <grabbing -ac7>
Quoted 5 lines
Quoted 1 line
OK, any brave soul to test that? All I can promise that it builds.
This looks like it would create a dummy block even for non-broken filesystems (ie truncating a file to be larger on ext2 would create a block, no?). While that would work, it would also waste disk-space.
Another idea is to create the hole for new_file_size + [one block], and then truncate that block off the end of the file, leaving nothing but the hole. I'll never admit to suggesting it though.
-chris
ftruncate not extending files?
In reply to Peter DaumQuoted 5 lines · Alan Cox
Quoted 2 lines
In that case, why was it changed for FAT only? Ext2 will still happily enlarge a file by truncating it.
ftruncate()andtruncate()may extend a file but they are not required to do so.
Stevens' example code assumes that it does. And given the number of people that regard his work as Holy, it might not be a bad idea to implement Linux so that it does what people think it does.
I would've sworn, based on the fact that I saw people do it, that ftruncate was a legitimate way to extend a file - especially useful in combination with mmap().
I don't really care where it is done, in glibc or in the kernel - but let's honor this convention and not needlessly break code.
Regards,
bert hubert
Signature · 3 lines
http://www.PowerDNS.com Versatile DNS Services Trilab The Technology People 'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
Re: ftruncate not extending files?
Quoted 6 lines · bert hubert
Quoted 2 lines
ftruncate()andtruncate()may extend a file but they are not required to do so.I would've sworn, based on the fact that I saw people do it, that ftruncate was a legitimate way to extend a file - especially useful in combination with
mmap().
lseek and write does it already and need not to duplicate with truncate(). Using truncate() to extend a file sounds very strange.
About mmap() SUSv2 says:
If the size of the mapped file changes after the call to mmap() as a result of some other operation on the mapped file, the effect of references to portions of the mapped region that correspond to added or removed portions of the file is unspecified.
^^^^^^^^^^^
What do you mean about "useful in combination with mmap()" ?
Quoted 2 lines
I don't really care where it is done, in glibc or in the kernel - but let's honor this convention and not needlessly break code.
Let's don't connive to ill-formed programs.
--
Re: ftruncate not extending files?
Quoted 18 lines · Dmitry A. Fedorov
Quoted 6 lines · bert hubert
Quoted 2 lines
ftruncate()andtruncate()may extend a file but they are not required to do so.I would've sworn, based on the fact that I saw people do it, that ftruncate was a legitimate way to extend a file - especially useful in combination with
mmap().lseek and write does it already and need not to duplicate with
truncate(). Usingtruncate()to extend a file sounds very strange.About
mmap()SUSv2 says:If the size of the mapped file changes after the call to
mmap()as a result of some other operation on the mapped file, the effect of references to portions of the mapped region that correspond to added or removed portions of the file is unspecified.
Quoted 7 lines
^^^^^^^^^^^What do you mean about "useful in combination with
mmap()" ?Quoted 2 lines
I don't really care where it is done, in glibc or in the kernel - but let's honor this convention and not needlessly break code.
Let's don't connive to ill-formed programs.
Those are not ill-formed programs, they are just ordinary programs.
ftruncate()-to-extend and mmap() go together like knife and fork:
you don't have to use either, and you can use each separately, but it's conventional to use them together. The awkward SIGBUS-beyond-EOF specification of mmap() begs for a call to set filesize; often the man page of one syscall references the other.
It would be perverse and unhelpful for Linux to offer a filesystem which supports mmap() but not ftrunctate()-to-extend.
The SuSv2 quotations, above and in other mail, are just weasly. They probably started off specifying the natural least-surprise behaviours, which most support; but some vendor who failed to meet that spec forced the wording to be changed to "unspecified". That's no reason for Linux to be obstinate.
But I agree "ftruncate" is a confusing name for a syscall to (truncate or) extend a file: like using a knife to spread glue.
Hugh
Re: ftruncate not extending files?
[email protected] (Hugh Dickins) wrote on 02.03.01 in <[email protected]>:
Quoted 1 line
The SuSv2 quotations, above and in other mail, are just weasly.
The next version is less weasly. Right now it's still a draft; what it says in draft 5 is this (note the markers which show what's optional to which feature set; XSI aka SUS-next-generation _does_ require expansion):
15245 NAME
15246 ftruncate - truncate a file to a specified length
15247 SYNOPSIS
15248 #include <unistd.h>
15249 int ftruncate(int fildes, off_t length);
15250 DESCRIPTION
15251 If fildes is not a valid file descriptor open for writing, the ftruncate( ) function shall fail. 15252 If fildes refers to a regular file, the ftruncate( ) function shall cause the size of the file to be 15253 truncated to length. If the size of the file previously exceeded length, the extra data shall no 15254 longer be available to reads on the file. If the file previously was smaller than this size, 15255 XSI ftruncate( ) shall either increase the size of the file or fail. XSI-conformant systems shall increase 15256 the size of the file. If the file size is increased, the extended area shall appear as if it were zero- | 15257 filled. The value of the seek pointer shall not be modified by a call to ftruncate( ). | 15258 Upon successful completion, if fildes refers to a regular file, the ftruncate( ) function shall mark 15259 for update the st_ctime and st_mtime fields of the file and the S_ISUID and S_ISGID bits of the file 15260 mode may be cleared. If the ftruncate( ) function is unsuccessful, the file is unaffected. 15261 XSI If the request would cause the file size to exceed the soft file size limit for the process, the 15262 request shall fail and the implementation shall generate the SIGXFSZ signal for the thread. | 15263 If fildes refers to a directory, ftruncate( ) shall fail. 15264 If fildes refers to any other file type, except a shared memory object, the result is unspecified. 15265 SHM If fildes refers to a shared memory object, ftruncate( ) shall set the size of the shared memory 15266 object to length. 15267 MF|SHM If the effect of ftruncate( ) is to decrease the size of a shared memory object or memory mapped 15268 file and whole pages beyond the new end were previously mapped, then the whole pages 15269 beyond the new end shall be discarded. 15270 MPR If the Memory Protection option is supported, references to discarded pages shall result in the 15271 generation of a SIGBUS signal; otherwise, the result of such references is undefined.
15272 MF|SHM If the effect of ftruncate( ) is to increase the size of a shared memory object, it is unspecified if the
15273 contents of any mapped pages between the old end-of-file and the new are flushed to the
15274 underlying object.
15275 RETURN VALUE
15276 Upon successful completion, ftruncate( ) shall return 0; otherwise, -1 shall be returned and errno
15277 set to indicate the error.
15278 ERRORS
15279 The ftruncate( ) function shall fail if: 15280 [EINTR] A signal was caught during execution. 15281 [EINVAL] The length argument was less than 0. 15282 [EFBIG] or [EINVAL] 15283 The length argument was greater than the maximum file size. 15284 XSI [EFBIG] The file is a regular file and length is greater than the offset maximum 15285 established in the open file description associated with fildes. 15286 [EIO] An I/O error occurred while reading from or writing to a file system. 15287 [EBADF] or [EINVAL] 15288 The fildes argument is not a file descriptor open for writing. 15289 [EINVAL] The fildes argument references a file that was opened without write 15290 permission. 15291 [EROFS] The named file resides on a read-only file system.
15292 EXAMPLES
15293 None.
15294 APPLICATION USAGE
15295 None.
15296 RATIONALE
15297 The ftruncate( ) function is part of IEEE Std 1003.1-200x as it was deemed to be more useful than |
15298 truncate( ). The truncate( ) function is provided as an XSI extension.
15299 FUTURE DIRECTIONS
15300 None.
15301 SEE ALSO
15302 open( ), truncate( ), the Base Definitions volume of IEEE Std 1003.1-200x, <unistd.h> |
15303 CHANGE HISTORY
15304 First released in Issue 4, Version 2.
15305 Issue 5
15306 Moved from X/OPEN UNIX extension to BASE and aligned with ftruncate( ) in the POSIX 15307 Realtime Extension. Specifically, the DESCRIPTION is extensively reworded and [EROFS] is 15308 added to the list of mandatory errors that can be returned by ftruncate( ). 15309 Large File Summit extensions are added. 15310 Issue 6 15311 The truncate( ) function has been split out into a separate reference page. 15312 The following new requirements on POSIX implementations derive from alignment with the 15313 Single UNIX Specification:
15314 * The DESCRIPTION is change to indicate that if the file size is changed, and if the file is a
15315 regular file, the S_ISUID and S_ISGID bits in the file mode may be cleared.
15316 The following changes were made to align with the IEEE P1003.1a draft standard:
15317 * The DESCRIPTION text is updated.
15318 XSI-conformant systems are required to increase the size of the file if the file was previously
15319 smaller than the size requested.
You can get the whole draft by subscribing to the mailing list at
http://www.opengroup.org/austin/.
MfG Kai
Re: ftruncate not extending files?
In reply to Peter DaumQuoted 2 lines · bert hubert
I would've sworn, based on the fact that I saw people do it, that ftruncate was a legitimate way to extend a file
Well it's not SuSv2 standards compliant:
http://www.opengroup.org/onlinepubs/007908799/xsh/ftruncate.html
If the file previously was larger than length, the extra data is discarded. If it was previously shorter than length, it is unspecified whether the file is changed or its size increased. If ^^^^^^^^^^^
the file is extended, the extended area appears as if it were zero-filled.
How "legitimate" relates to "SuSv2 standards compliant" is your call.
--Malcolm
Signature · 3 lines
Malcolm Beattie <[email protected]> Unix Systems Programmer Oxford University Computing Services