64-bit block sizes on 32-bit systems
--------- Received message begins Here ---------
Quoted 13 lines · LA Walsh
Quoted 11 lines
Now lets look at the sites want to process terabytes of
data -- perhaps files systems up into the Pentabyte range. Often I can see these being large multi-node (think 16-1024 clusters as are in use today for large super-clusters). If I was to characterize the performance of them, I'd likely see the CPU pegged at 100% with 99% usage in user space. Let's assume that increasing the block size decreases disk accesses by as much as 10% (you'll have to admit -- using a 64bit quantity vs. 32bit quantity isn't going to even come close to increasing disk access times by 1 millisecond, really, so it really is going to be a much smaller fraction when compared to the actual disk latency).
Quoted 8 lines
[snip]
Quoted 1 line
Is there some logical flaw in the above reasoning?
But those changes will affect even the fastpath, i.e. data that is already in the page/buffer caches. In which case we don't have to wait for disk access latency. Why would anyone who is working with a pentabyte of data even consider not relying on essentially always hitting data that is available the read-ahead cache.
It depends entirely on the application. Where the cache can contain 20% of the data, most accesses should already be in memory. If the data is significantly larger, there is a high chance that the data will not be there.
Quoted 6 lines
Using similar numbers as presented. If we are working our way through every single block in a Pentabyte filesystem, and the blocksize is 512 bytes. Then the 1us in extra CPU cycles because of 64-bit operations would add, according to by back of the envelope calculation, 2199023 seconds of CPU time a bit more than 25 days.
Ummm... I don't think it adds that much. You seem to be leaving out the overlap disk/IO and computation for read-ahead. This should eliminate the majority of the delay effect.
Quoted 7 lines
Seriously, there is a lot more that needs to be done than introducing a 64-bit blocknumber. Effectively 512 byte blocks are far too small for that kind of data, and going to pagesize blocks (and increasing pagesize to 64KB or 2MB at the same time) is a solution that is far more likely to give good results since it reduces both the total the number of 'blocks' on the device as well as reducing the total amount of calls throughout kernel space instead of increasing the cost per call.
Talk about adding overhead... How long do you think it takes to read a 2MB block (not to mention the time to update that page..) The additional contention on the fiberchannel I/O alone might kill it if the filesystem is busy.
Granted, 512 bytes could be considered too small for some things, but once you pass 32K you start adding a lot of rotational delay problems. I've used file systems with 256K blocks - they are slow when compaired to the throughput using 32K. I wasn't the one running the benchmarks, but with a MaxStrat 400GB raid with 256K sized data transfer was much slower (around 3 times slower) than 32K. (The target application was a GIS server using Oracle).
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Quoted 9 lines · Jesse Pollard
Quoted 5 lines
Using similar numbers as presented. If we are working our way through every single block in a Pentabyte filesystem, and the blocksize is 512 bytes. Then the 1us in extra CPU cycles because of 64-bit operations would add, according to by back of the envelope calculation, 2199023 seconds of CPU time a bit more than 25 days.
Ummm... I don't think it adds that much. You seem to be leaving out the overlap disk/IO and computation for read-ahead. This should eliminate the majority of the delay effect.
1024 TB should be around 2*10^12 512-byte blocks, divide by 10^6 (1us) of "assumed" overhead per block operation is 2*10^6 seconds, no I believe I'm pretty close there. I am considering everything being "available in the cache", i.e. no waiting for disk access.
Quoted 8 lines
Quoted 7 lines
Seriously, there is a lot more that needs to be done than introducing a 64-bit blocknumber. Effectively 512 byte blocks are far too small for that kind of data, and going to pagesize blocks (and increasing pagesize to 64KB or 2MB at the same time) is a solution that is far more likely to give good results since it reduces both the total the number of 'blocks' on the device as well as reducing the total amount of calls throughout kernel space instead of increasing the cost per call.
Quoted 4 lines
Talk about adding overhead... How long do you think it takes to read a 2MB block (not to mention the time to update that page..) The additional contention on the fiberchannel I/O alone might kill it if the filesystem is busy.
The time to update the pagetables is identical to the time to update a 4KB page when the OS is using a 2MB pagesize. Ofcourse it will take more time to load the data into the page, however it should be a consecutive stretch of data on disk, which should give a more efficient transfer than small blocks scattered around the disk.
Quoted 7 lines
Granted, 512 bytes could be considered too small for some things, but once you pass 32K you start adding a lot of rotational delay problems. I've used file systems with 256K blocks - they are slow when compaired to the throughput using 32K. I wasn't the one running the benchmarks, but with a MaxStrat 400GB raid with 256K sized data transfer was much slower (around 3 times slower) than 32K. (The target application was a GIS server using Oracle).
But your subsystem (the disk) was probably still using 512 byte blocks, possibly scattered. And the OS was still using 4KB pages, it takes more time to reclaim and gather 64 pages per IO operation than one, that's why I'm saying that the pagesize needs to scale along with the blocksize.
The application might have been assuming a small block size as well, and the OS was told to do several read/modify/write cycles, perhaps even 512 times as much as necessary.
I'm not saying that the current system will perform well when working with large blocks, but compared to increasing the size of block_t, a larger blocksize has more potential to give improvements in the long term without adding an unrecoverable performance hit.
Jan
Quoted 8 lines · Jan Harkes
Quoted 6 lines · Jesse Pollard
Quoted 5 lines
Using similar numbers as presented. If we are working our way through every single block in a Pentabyte filesystem, and the blocksize is 512 bytes. Then the 1us in extra CPU cycles because of 64-bit operations would add, according to by back of the envelope calculation, 2199023 seconds of CPU time a bit more than 25 days.
Quoted 8 lines
Quoted 3 lines
Ummm... I don't think it adds that much. You seem to be leaving out the overlap disk/IO and computation for read-ahead. This should eliminate the majority of the delay effect.
1024 TB should be around 2*10^12 512-byte blocks, divide by 10^6 (1us) of "assumed" overhead per block operation is 2*10^6 seconds, no I believe I'm pretty close there. I am considering everything being "available in the cache", i.e. no waiting for disk access.
---
If everything being used is only used from the cache, then
the application probably doesn't need 64-bit block support.
I submit that your argument may be flawed in the assumption that
if an application needs multi-terabyte files and devices, that most of the data will be in the in-memory cache.
Quoted 5 lines
The time to update the pagetables is identical to the time to update a 4KB page when the OS is using a 2MB pagesize. Ofcourse it will take more time to load the data into the page, however it should be a consecutive stretch of data on disk, which should give a more efficient transfer than small blocks scattered around the disk.
---
Not if you were doing alot of random reads where you only
needd 1-2K of data. The read-time of the extra 2M-1K would seem to eat into any performance boot gained by the large pagesize.
Quoted 14 lines
Quoted 8 lines
Granted, 512 bytes could be considered too small for some things, but once you pass 32K you start adding a lot of rotational delay problems. I've used file systems with 256K blocks - they are slow when compaired to the throughput using 32K. I wasn't the one running the benchmarks, but with a MaxStrat 400GB raid with 256K sized data transfer was much slower (around 3 times slower) than 32K. (The target application was a GIS server using Oracle).
But your subsystem (the disk) was probably still using 512 byte blocks, possibly scattered. And the OS was still using 4KB pages, it takes more time to reclaim and gather 64 pages per IO operation than one, that's why I'm saying that the pagesize needs to scale along with the blocksize.
Quoted 8 lines
The application might have been assuming a small block size as well, and the OS was told to do several read/modify/write cycles, perhaps even 512 times as much as necessary.
I'm not saying that the current system will perform well when working with large blocks, but compared to increasing the size of
block_t, a larger blocksize has more potential to give improvements in the long term without adding an unrecoverable performance hit.
---
That's totally application dependent. Database applications
might tend to skip around in the data and do short/reads/writes over a very large file. Large block sizes will degrade their performance.
This was the idea of making it a *configurable* option. If
you need it, configure it. Same with block size -- that should likely have a wider range for configuration as well. But configuration (and ideally auto-configuration where possible) seems the ultimate win-win situation.
-l
Signature · 4 lines
The above thoughts are my own and do not necessarily represent those of my employer. L A Walsh | Trust Technology, Core Linux, SGI [email protected] | Voice: (650) 933-5338