files of size larger than fs size
Hello!
I've discovered that it is possible to create files of size much larger than partition size.
I thought that this is JFS bug, so I've filed a bugreport against it at http://bugzilla.kernel.org/show_bug.cgi?id=4345
Detailed info and testcase program are provided there.
Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel.
Could kernel gurus please investigate the problem?
P.S. I'm using debian-amd64 with kernel 2.6.11.3. So I didn't check if 32-bit kernel has this bug.
Max
Quoted 10 lines · Max
Hello!
I've discovered that it is possible to create files of size much larger than partition size.
I thought that this is JFS bug, so I've filed a bugreport against it at http://bugzilla.kernel.org/show_bug.cgi?id=4345
Detailed info and testcase program are provided there.Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel.
Could kernel gurus please investigate the problem?
Your test case isn't writing a full file, it is only writing 4 bytes at various offsets (1^32, 1^40, 1^48, 1^56).
The filesystems you mention support files with "holes" in them, in otherwords they support gaps between data which don't take up any storage.
Even though your test case is creating a huge file, only a couple of bytes are written, the rest of the huge file doesn't take up any space.
The behaviour you're seeing isn't a bug...
Phillip Lougher
Quoted 8 lines · Max
Hello!
I've discovered that it is possible to create files of size much larger than partition size.
I thought that this is JFS bug, so I've filed a bugreport against it at
http://bugzilla.kernel.org/show_bug.cgi?id=4345
Detailed info and testcase program are provided there.
Quoted 8 lines
Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel.
Could kernel gurus please investigate the problem?
P.S. I'm using debian-amd64 with kernel 2.6.11.3. So I didn't check if 32-bit kernel has this bug.
This is not a bug but a fairly standard UNIX feature where you are creating sparse files. If you stat these files, you'll see the actual number of filesystem blocks allocated is much smaller than the apparent length of the file.
For more info: google.com
This one looks like a decent overview:
http://www.lrdev.com/lr/unix/sparsefile.html
Sonny
Thanks for the explanation.
I closed my bugreport as INVALID.
But anyway it's ineteresting why the resulting sparse files have different size on different fs?
Max
Quoted 30 lines · Phillip Lougher
Quoted 10 lines · Max
Hello!
I've discovered that it is possible to create files of size much larger than partition size.
I thought that this is JFS bug, so I've filed a bugreport against it at http://bugzilla.kernel.org/show_bug.cgi?id=4345
Detailed info and testcase program are provided there.Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel.
Could kernel gurus please investigate the problem?
Your test case isn't writing a full file, it is only writing 4 bytes at various offsets (1^32, 1^40, 1^48, 1^56).
The filesystems you mention support files with "holes" in them, in otherwords they support gaps between data which don't take up any storage.
Even though your test case is creating a huge file, only a couple of bytes are written, the rest of the huge file doesn't take up any space.
The behaviour you're seeing isn't a bug...
Phillip Lougher
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
But anyway it's interesting why the resulting sparse files have different size on different fs?
That looks like a bug. Assuming you didn't see any seeks or writes fail, the file size on all filesystems should be 2^56 + 4. I suspect this is beyond the maximum file size allowed by the filesystem in some cases, so the write isn't happening, which means you should get a failure return code.
In the results you showed, the filesize ends up being a little less than 2^48, which is not a place that you wrote ever.
--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems
Bryan,
I'm not experienced in filesystems. But I've derived the testcase program from some buggy application that occasionally created huge files on my fs. That was not so easy to reproduce since not every sequence of seeks/reads/writes results in a huge file. But finally I got it 100% reproducible.
So I would appriciate if somebody make any good from that testcase code.
Different filesizes look strange to me. What I have so far:
JFS: 281474976706576 = 2**48 - 4080 XFS: 72057594037923856 = 2**56 - 4080 EXT3: 1099511627784 = 2**40 + 8
Thanks,
Max
P.S. direct link to the testcase program: http://bugzilla.kernel.org/attachment.cgi?id=4729
Quoted 12 lines · Bryan Henderson
Quoted 1 line
But anyway it's interesting why the resulting sparse files have different size on different fs?
That looks like a bug. Assuming you didn't see any seeks or writes fail, the file size on all filesystems should be 2^56 + 4. I suspect this is beyond the maximum file size allowed by the filesystem in some cases, so the write isn't happening, which means you should get a failure return code.
In the results you showed, the filesize ends up being a little less than 2^48, which is not a place that you wrote ever.
--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems
Quoted 10 lines · Max
Bryan,
I'm not experienced in filesystems. But I've derived the testcase program from some buggy application that occasionally created huge files on my fs. That was not so easy to reproduce since not every sequence of seeks/reads/writes results in a huge file. But finally I got it 100% reproducible.
So I would appriciate if somebody make any good from that testcase code.
man lseek tells you to test the return code against (off_t)-1. I found that for larger values, your test program is returning -1, but unsigned it appears as 18446744073709551615.
Quoted 6 lines
Different filesizes look strange to me. What I have so far:
JFS: 281474976706576 = 2**48 - 4080 XFS: 72057594037923856 = 2**56 - 4080 EXT3: 1099511627784 = 2**40 + 8
I can speak for jfs, and it supports file sizes up to 2**52. It uses 40 bits to address the logical block offset within a file with a 4K (2**12) size block. The individual file systems set sb->s_maxbytes to the appropriate value.
Quoted 4 lines
Thanks,
Max
P.S. direct link to the testcase program: http://bugzilla.kernel.org/attachment.cgi?id=4729
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Quoted 3 lines
I found
that for larger values, your test program is returning -1, but unsigned it appears as 18446744073709551615.
You mean you ran it? Then what about the more interesting question of what your filesize ends up to be? You say JFS allows files up to 2**52 bytes, so I expect the test case would succeed up through the write at 2**48 and leave the filesize 2**48 + 8. But Max reports seeing 2**48 - 4080.
It's conceivable that the reporting of the filesize is wrong, by the way.
Quoted 9 lines · Bryan Henderson
Quoted 3 lines
I found
that for larger values, your test program is returning -1, but unsigned it appears as 18446744073709551615.You mean you ran it? Then what about the more interesting question of what your filesize ends up to be? You say JFS allows files up to 2**52 bytes, so I expect the test case would succeed up through the write at 2**48 and leave the filesize 2**48 + 8. But Max reports seeing 2**48 - 4080.
Yeah, this is funny. I missed that part of the story. I'll try to figure out what is going on here.
Quoted 1 line
It's conceivable that the reporting of the filesize is wrong, by the way.
jfs_debugfs tells me the same thing. So ls has it right.
Thanks,
Shaggy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Quoted 5 lines · Dave Kleikamp
Quoted 4 lines · Bryan Henderson
Quoted 3 lines
I found
that for larger values, your test program is returning -1, but unsigned it appears as 18446744073709551615.
Quoted 8 lines
Quoted 5 lines
You mean you ran it? Then what about the more interesting question of what your filesize ends up to be? You say JFS allows files up to 2**52 bytes, so I expect the test case would succeed up through the write at 2**48 and leave the filesize 2**48 + 8. But Max reports seeing 2**48 - 4080.
Yeah, this is funny. I missed that part of the story. I'll try to figure out what is going on here.
The problem appears to be mixing calls to lseek64 with calls to fread and fwrite. fread & fwrite don't appear to honor the position set by lseek64. I assume replacing fread & fwrite with read & write will fix it. I haven't tried it yet, but I'm about to.
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
With this change, the file size on jfs becomes 2^48 + 4 as expected.
jfs_bug.c renamed from jfs_bug.c.orig +4 -2
@@ -13,12 +13,14 @@ int data = 0;struct flock fl;void read1 () { -size_t rc =fread (&data,sizeof (data),1 ,f); +/* size_t rc = fread(&data,sizeof(data),1,f); */ +size_t rc =read (fn, &data,sizeof (data));printf ("read() rc = %llu\n" ,rc); }void write1 () { -size_t rc =fwrite (&data,sizeof (data),1 ,f); +/* size_t rc = fwrite(&data,sizeof(data),1,f); */ +size_t rc =write (fn, &data,sizeof (data));printf ("write() rc = %llu\n" ,rc); }
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Dave,
Shouldn't "fread(&data,sizeof(data),1,f)" and "read(fn, &data, sizeof(data))" produce identical results? Is it a bug or what?
Max
Quoted 24 lines · Dave Kleikamp
With this change, the file size on jfs becomes 2^48 + 4 as expected.
jfs_bug.c renamed from jfs_bug.c.orig +4
-2@@ -13,12 +13,14 @@ int data = 0;struct flock fl;void read1 () { -size_t rc =fread (&data,sizeof (data),1 ,f); +/* size_t rc = fread(&data,sizeof(data),1,f); */ +size_t rc =read (fn, &data,sizeof (data));printf ("read() rc = %llu\n" ,rc); }void write1 () { -size_t rc =fwrite (&data,sizeof (data),1 ,f); +/* size_t rc = fwrite(&data,sizeof(data),1,f); */ +size_t rc =write (fn, &data,sizeof (data));printf ("write() rc = %llu\n" ,rc); }
Quoted 5 lines · Max
Dave,
Shouldn't "fread(&data,sizeof(data),1,f)" and "
read(fn, &data, sizeof(data))" produce identical results?
Is it a bug or what?
fseek(f, offset, SEEK_SET);
fread(&data, sizeof(data, 1, f);
should produce identical results to
lseek(fn, offset, SEEK_SET);
read(fn, &data, sizeof(data);
However
lseek(fn, offset, SEEK_SET);
fread(&data, sizeof(data, 1, f);
isn't valid.
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Quoted 2 lines
The problem appears to be mixing calls to lseek64 with calls to fread and fwrite.
Oh, of course. I didn't see that. You can't use the file descriptor of a file that is opened as a stream. This test case uses the fileno() function to mess with the internals of the stream.
fseeko64() is the proper function to position a stream.