64-bit block sizes on 32-bit systems
Jan Harkes <[email protected]>:
Quoted 8 lines · Jesse Pollard
Quoted 6 lines
Quoted 5 lines
Using similar numbers as presented. If we are working our way through every single block in a Pentabyte filesystem, and the blocksize is 512 bytes. Then the 1us in extra CPU cycles because of 64-bit operations would add, according to by back of the envelope calculation, 2199023 seconds of CPU time a bit more than 25 days.
Quoted 8 lines
Quoted 3 lines
Ummm... I don't think it adds that much. You seem to be leaving out the overlap disk/IO and computation for read-ahead. This should eliminate the majority of the delay effect.
1024 TB should be around 2*10^12 512-byte blocks, divide by 10^6 (1us) of "assumed" overhead per block operation is 2*10^6 seconds, no I believe I'm pretty close there. I am considering everything being "available in the cache", i.e. no waiting for disk access.
That would be true for small files (< 5GB). I have to deal with files that may be 20-100 GB. Except for the largest systems (200GB of main memory) the data will NOT be in the cache except for ~50% of the time. (assuming only one user....)
Quoted 13 lines
Quoted 8 lines
Quoted 7 lines
Seriously, there is a lot more that needs to be done than introducing a 64-bit blocknumber. Effectively 512 byte blocks are far too small for that kind of data, and going to pagesize blocks (and increasing pagesize to 64KB or 2MB at the same time) is a solution that is far more likely to give good results since it reduces both the total the number of 'blocks' on the device as well as reducing the total amount of calls throughout kernel space instead of increasing the cost per call.
Quoted 4 lines
Talk about adding overhead... How long do you think it takes to read a 2MB block (not to mention the time to update that page..) The additional contention on the fiberchannel I/O alone might kill it if the filesystem is busy.
Quoted 5 lines
The time to update the pagetables is identical to the time to update a 4KB page when the OS is using a 2MB pagesize. Ofcourse it will take more time to load the data into the page, however it should be a consecutive stretch of data on disk, which should give a more efficient transfer than small blocks scattered around the disk.
You assume the file is accessed sequentially. The wether models don't do that. They do have some locality, but only in a 3D sense. When you include time it becomes closer to a random disk block reference when everything has to be linearized.
Quoted 9 lines
Quoted 8 lines
Granted, 512 bytes could be considered too small for some things, but once you pass 32K you start adding a lot of rotational delay problems. I've used file systems with 256K blocks - they are slow when compaired to the throughput using 32K. I wasn't the one running the benchmarks, but with a MaxStrat 400GB raid with 256K sized data transfer was much slower (around 3 times slower) than 32K. (The target application was a GIS server using Oracle).
Quoted 4 lines
But your subsystem (the disk) was probably still using 512 byte blocks, possibly scattered. And the OS was still using 4KB pages, it takes more time to reclaim and gather 64 pages per IO operation than one, that's why I'm saying that the pagesize needs to scale along with the blocksize.
It wasn't - the "disks" were composed of groups of 5 drives in a raid striped for speed and spread across 5 SCSI III controlers. Each raid attached had 16MB internal cache. I think the controlers were using an entire sector read (32K).
Quoted 3 lines
The application might have been assuming a small block size as well, and the OS was told to do several read/modify/write cycles, perhaps even 512 times as much as necessary.
There was some of that, but not much. Oracle (as I recall) allows for the specification of transfer size.
This also brings up the problem of small files. Allocating 2MB per file would, waist quite a bit of disk space (assuming 5 - 10 million files with only 15% having 25GB or more).
Quoted 4 lines
I'm not saying that the current system will perform well when working with large blocks, but compared to increasing the size of
block_t, a larger blocksize has more potential to give improvements in the long term without adding an unrecoverable performance hit.
Not when the filesystem is required for general use. It only makes it simpler to actually have a large filesystem. It doesn't help when it must be used.
Now you are saying that the throughput WILL go down, but only if you use large block sizes.
I can go along with making block sizes up to 8K. Even 32K for special circumstances (even 64K for dedicated use). But not larger. NFS overhead on file I/O becomes way too excessive (...worst example now is having to read a 2MB block to update 512 bytes, then write it back... :-)
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Hi,
Just a brief add to the discussion, besides which I have a vested interest in this!
I do not believe that you can make the addressability of a device larger at the expense of granularity of address space at the bottom end. Just because ext2 has a single size for metadata does not mean everything you put on the disks does. XFS filesystems, for example, can be made with block sizes from 512 bytes to 64Kbytes (ok not working on linux across this range yet, but it will).
In all of these cases we have chunks of metadata which are 512 bytes long, and we have chunks bigger than the blocksize. The 512 byte chunks are the superblock and the heads of the freespace structures, there are multilples of them through the filesystem.
To top that, we have disk write ordering constraints that could mean that for two of the 512 byte chunks next to each other one must be written to disk now to free log space, the other must not be written to disk because it is in a transaction. We would be forced to do read-modify-write down at some lower level - wait the lower levels would not have the addressability.
There are probably other things which will not fly if you lose the addressing granularity. Volume headers and such like would be one possibility.
No I don't have a magic bullet solution, but I do not think that just increasing the granularity of the addressing is the correct answer, and yes I do agree that just growing the buffer_head fields is not perfect either.
Steve Lord
p.s. there was mention of bigger page size, it is not hard to fix, but the swap path will not even work with 64K pages right now.
Quoted 2 lines · Steve Lord
Just a brief add to the discussion, besides which I have a vested interest in this!
I'll add my little comments as well, and hopefully not start a flamewar... :)
[snip comments about blocksize, etc.]
Here's a real-life example of something that most of you will probably hate me for mentioning:
HFS uses variable sized blocks (made up of multiple 512 byte sectors), but stores block numbers as a 16 bit value. (I know, everyone will say, "We're talking about moving from 32 to 64 bits." Keep listening.) This gave great performance on the then current massive storage of a 20M drive. However, when it became possible to get the absolutely gigantic hard drive of 1G, it became more and more obvious that it was a drawback that was causing a huge amount of wasted space. Apple had to design a new filesystem (HFS+) that was able to represent blocks with a 32 bit number to overcome the effective limitation on how big a filesystem could be. It's getting to the point now that it's easily possible to put together a disk array that is large enough that even referring to blocks with a 32 bit value requires relatively large blocks. I don't know if we have very many filesystems that would support this feature, but it will become important a lot sooner than anyone may be thinking.
Obviously this case isn't a perfect fit for the situation, since HFS was designed to be read by 32 bit machines, and the upgrade to 32 bits didn't give a CPU penalty, just a bus bandwidth problem. Also, I'm coming from a platform that actually can do a decent job of 64 bit, unlike x86, but we shouldn't disallow people from doing bigger and better things. It's become very popular lately to position Linux as an enterprise-ready system, and this is something that will be expected. People will want to access a multi-TB database as a single file, as well as other things that may seem crazy to most people now.
I understand people's aversion to the #ifdefs in the code, but if the changes are made in a sane way, it can still be clean and easy to maintain. It's worth it to add a little complexity (particularly as an option) to add a feature that people will be demanding in the relatively near future. It might be a good idea to wait for 2.5, tho...
Brad Boyer
[email protected]
P.S.: No, I have no personal reason to need any of this 64 bit filesystem stuff. Just trying to point out possibilities. Don't expect me to actually be writing this stuff...
My turn to chime in.
JFS was designed around a 4K meta-data page size. It would require some major re-design to use larger block sizes. On the other hand, JFS could take advantage of 64-bit block addresses immediately. JFS internally store the block address in 40 bits. (Sorry, file size & volume size are both limited to 4 peta-bytes on JFS.)
At the rate that storage hardware and requirements are increasing, increasing the block size is a short-term solution that is only going to delay the inevitable requirement for 64-bit block addressability. There is a practical limit to a usable block-size. Someone threw out 64K, which seems reasonable to me.
Signature · 2 lines
David Kleikamp IBM Linux Technology Center