Large-file corruption. ReiserFS? VFS?
I ran mkisofs (2.01a16) this morning on a 2.6.0-test4 machine with the target on a 60GB ReiserFS partition. The resulting file was to be just over 4GB (around 4,360,000,000 bytes), and during most of the course of writing the file, a small prefix looked as it should through a hexdump: that is, just part of the iso9660 directory.
However, just as the write completed, the beginning of the file became corrupted. I considered a 4GB problem to be likely, and re-tested with fewer source files such that the result would be smaller than 4GB; lo and behold, no corruption. The same result occurs whether mkisofs is given the -o flag, or output is redirected to a file from stdout using the shell's redirection facility, suggesting the problem is probably at the kernel level.
I don't have a large enough ext2/3 filesystem to compare with, so there's no easy way for me to tell whether this is a Reiser-only problem or not. Can anyone confirm that they see the same problem, or whether they see a similar problem on another file system? Please feel free to ask me for any other information you think might be illuminating.
Cheers,
Kyle
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines
I would guess that mkisofs isn't opening the file with
O_LARGEFILE. It probably only expected to write 600MB output files. Purely a guess though.
Thanks for the suggestion, but it is, in fact, opening with O_LARGEFILE:
open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
Cheers,
Kyle
Quoted 8 lines · Kyle Rose
Quoted 3 lines
I would guess that mkisofs isn't opening the file with
O_LARGEFILE. It probably only expected to write 600MB output files. Purely a guess though.Thanks for the suggestion, but it is, in fact, opening with
O_LARGEFILE:
open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666)= 3
True, and O_LARGEFILE would have bit you at 2GB and not 4GB... If you are doing output redirected from the shell, then it can't be a seek issue either.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 3 lines · Andreas Dilger
True, and
O_LARGEFILEwould have bit you at 2GB and not 4GB... If you are doing output redirected from the shell, then it can't be a seek issue either.
Nah! Some programs just "try to seek" and if it works, will.
dd if=/dev/zero count=200 | strace sox -t raw -r 44100 -s -w - -t wav - > /tmp/test
dd if=/dev/zero count=200 | sox -t raw -r 44100 -s -w - -t wav - | dd of=/tmp/test
sox: Length in output .wav header will be wrong since can't seek to fix it
Roger.
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* **** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
Quoted 5 lines · Andreas Dilger
Quoted 4 lines · Kyle Rose
Quoted 3 lines
I would guess that mkisofs isn't opening the file with
O_LARGEFILE. It probably only expected to write 600MB output files. Purely a guess though.
Quoted 8 lines
Quoted 4 lines
Thanks for the suggestion, but it is, in fact, opening with
O_LARGEFILE:
open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666)= 3True, and
O_LARGEFILEwould have bit you at 2GB and not 4GB... If you are doing output redirected from the shell, then it can't be a seek issue either.
... but mkisofs will NOT be seeking, as we commonly burn CDs while mkisofs is providing the data through a pipe.
Roger.
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* **** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
Quoted 7 lines · Kyle Rose
I don't have a large enough ext2/3 filesystem to compare with, so there's no easy way for me to tell whether this is a Reiser-only problem or not. Can anyone confirm that they see the same problem, or whether they see a similar problem on another file system? Please feel free to ask me for any other information you think might be illuminating.
I can only speak for 2.4-Kernel
I burn DVDs since more than 2 years now. Starting with 2.4.4 and i had NEVER problems with the size, i never used the split-option of mkisofs (and i don't even know the correct(tm) name of it. I only know that it exists).
The maximum is 4489MB, (>4.700.000.000 bytes) and the biggest images i had are only a few KB smaller that that.
Since about 2 month i use reiserfs on the "source" and on the "target"-HDD. No changes, except that reiserfs is slower then ext2.
Bis denn
Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous.
Quoted 1 line
I can only speak for 2.4-Kernel
Yes, I never had a problem doing this on 2.4. It started when I upgraded to 2.6.
Cheers,
Kyle
Hello!
Quoted 2 lines · Kyle Rose
However, just as the write completed, the beginning of the file became corrupted. I considered a 4GB problem to be likely, and re-tested
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description. The patch below should help. Please confirm that it works for you too. Thanks a lot for the report.
Bye,
Oleg
===== fs/reiserfs/file.c 1.22 vs edited =====
fs/reiserfs/file.c +3 -3
@@ -779,7 +779,7 @@/* Now if all the write area lies past the file end, no point in maping blocks, since there is none, so we just zero out remaining parts of first and last pages in write area (if needed) */ -if ( (pos & ~(PAGE_CACHE_SIZE -1 )) > inode->i_size ) { +if ( (pos & ~((loff_t )PAGE_CACHE_SIZE -1 )) > inode->i_size ) {if ( from !=0 ) {/* First page needs to be partially zeroed */ char *kaddr =kmap_atomic (prepared_pages[0 ],KM_USER0 );memset (kaddr,0 , from);
@@ -801,9 +801,9 @@we need to allocate (calculated above) */ /* Mask write position to start on blocksize, we do it out of the loop for performance reasons */ - pos &= ~(inode->i_sb->s_blocksize -1 ); + pos &= ~((loff_t ) inode->i_sb->s_blocksize -1 );/* Set cpu key to the starting position in a file (on left block boundary)*/ -make_cpu_key (&key, inode,1 + ((pos) & ~(inode->i_sb->s_blocksize -1 )),TYPE_ANY ,3 /*key length*/ ); +make_cpu_key (&key, inode,1 + ((pos) & ~((loff_t ) inode->i_sb->s_blocksize -1 )),TYPE_ANY ,3 /*key length*/ );reiserfs_write_lock (inode->i_sb);// We need that for at least search_by_key() for ( i =0 ; i < num_pages ; i++ ) {
Quoted 4 lines · Oleg Drokin
Hello!
On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:
Quoted 8 lines
Quoted 2 lines
However, just as the write completed, the beginning of the file became corrupted. I considered a 4GB problem to be likely, and re-tested
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description. The patch below should help. Please confirm that it works for you too. Thanks a lot for the report.
Yow, I guess large files on reiserfs in 2.6 isn't very common...
Hello!
Quoted 7 lines · Mike Fedyk
Quoted 6 lines
Quoted 2 lines
However, just as the write completed, the beginning of the file became corrupted. I considered a 4GB problem to be likely, and re-tested
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description. The patch below should help. Please confirm that it works for you too. Thanks a lot for the report.Yow, I guess large files on reiserfs in 2.6 isn't very common...
Or may be nobody noticed the corruption.
Bye,
Oleg
Quoted 5 lines · Oleg Drokin
Hello!
Quoted 2 lines · Mike Fedyk
Quoted 2 lines
Quoted 2 lines
However, just as the write completed, the beginning of the file became corrupted. I considered a 4GB problem to be likely, and re-tested
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description. The patch below should help. Please confirm that it works for you too. Thanks a lot for the report.Yow, I guess large files on reiserfs in 2.6 isn't very common...
Or may be nobody noticed the corruption.
Possible.
Does this affect 2.4 also? If not, then that will narrow the possible number of people who could have hit this bug.
Luckily when I was using reiserfs, I used mostly small files.
Quoted 1 line
Does this affect 2.4 also?
As I said, this did not affect 2.4, at least not uniformly: I regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2} without problems.
Cheers,
Kyle
Quoted 5 lines · Kyle Rose
Quoted 1 line
Does this affect 2.4 also?
As I said, this did not affect 2.4, at least not uniformly: I regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2} without problems.
Ok, thanks. I'm glad it wasn't in 2.4
EOST
Quoted 12 lines · Mike Fedyk
Quoted 8 lines · Kyle Rose
Quoted 3 lines
Does this affect 2.4 also?
As I said, this did not affect 2.4, at least not uniformly: I regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2} without problems.
Quoted 12 lines
Ok, thanks. I'm glad it wasn't in 2.4
EOST
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
It was a bug in Oleg's large write patch, which it seems was truly for 2.5 only.....
Signature · 1 line
Hans
Hello!
Quoted 11 lines · Mike Fedyk
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
However, just as the write completed, the beginning of the file became corrupted. I considered a 4GB problem to be likely, and re-tested
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description. The patch below should help. Please confirm that it works for you too. Thanks a lot for the report.Yow, I guess large files on reiserfs in 2.6 isn't very common...
Or may be nobody noticed the corruption.
Possible.
Does this affect 2.4 also? If not, then that will narrow the possible number of people who could have hit this bug.
No, 2.4 is not affected.
Bye,
Oleg
Quoted 4 lines
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description. The patch below should help. Please confirm that it works for you too. Thanks a lot for the report.
This indeed solves the problem. Thanks, Oleg!
Cheers,
Kyle