remove 2TB block device limit
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Daniel" == Daniel Phillips <[email protected]> writes:
Quoted 4 lines · Anton Altaparmakov
Quoted 4 lines
...And yes at the moment the pagecache limit is also a problem which we just ignore in the hope that the kernel will have gone to 64 bits by the time devices grow that large as to start using > 32 bits of blocks/pages...
Daniel> PAGE_CACHE_SIZE can also grow, so 32 bit architectures are Daniel> further away from the page cache limit on than it seems.
Check out the table on page 2 of
http://www.scsita.org/statech/01s005r1.pdf
The SCSI trade association is predicting 200TB in a high-end server within 10 years --- and 2TB in a high-end desktop by 2004. I'd take some of their predictions with a grain of salt, however.
Quoted 11 lines · Peter Chubb
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Daniel" == Daniel Phillips <[email protected]> writes:
Quoted 4 lines · Anton Altaparmakov
Quoted 4 lines
...And yes at the moment the pagecache limit is also a problem which we just ignore in the hope that the kernel will have gone to 64 bits by the time devices grow that large as to start using > 32 bits of blocks/pages...
Daniel>
PAGE_CACHE_SIZEcan also grow, so 32 bit architectures are Daniel> further away from the page cache limit on than it seems.
Quoted 6 lines
Check out the table on page 2 of
http://www.scsita.org/statech/01s005r1.pdfThe SCSI trade association is predicting 200TB in a high-end server within 10 years --- and 2TB in a high-end desktop by 2004. I'd take some of their predictions with a grain of salt, however.
The server definitely won't be running a 32 bit processor, and the high end desktop probably won't. In any event, the current 44 bit limitation (32 bit arch) on the page cache takes us up to 16 TB, and going to a 16 KB PAGE_CACHE_SIZE takes us to 64 TB, so I don't think we have to start slicing and dicing that part of the kernel just now. Anybody who expects to run into this limitation should of course raise their hand.
Incidently, the 200 TB high-end servers are a lot closer than you think.
Signature · 1 line
Daniel
Daniel Phillips <[email protected]>:
Quoted 19 lines · Peter Chubb
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Daniel" == Daniel Phillips <[email protected]> writes:
Quoted 4 lines · Anton Altaparmakov
Quoted 4 lines
...And yes at the moment the pagecache limit is also a problem which we just ignore in the hope that the kernel will have gone to 64 bits by the time devices grow that large as to start using > 32 bits of blocks/pages...
Daniel>
PAGE_CACHE_SIZEcan also grow, so 32 bit architectures are Daniel> further away from the page cache limit on than it seems.Quoted 6 lines
Check out the table on page 2 of
http://www.scsita.org/statech/01s005r1.pdfThe SCSI trade association is predicting 200TB in a high-end server within 10 years --- and 2TB in a high-end desktop by 2004. I'd take some of their predictions with a grain of salt, however.
Quoted 8 lines
The server definitely won't be running a 32 bit processor, and the high end desktop probably won't. In any event, the current 44 bit limitation (32 bit arch) on the page cache takes us up to 16 TB, and going to a 16 KB
PAGE_CACHE_SIZEtakes us to 64 TB, so I don't think we have to start slicing and dicing that part of the kernel just now. Anybody who expects to run into this limitation should of course raise their hand.Incidently, the 200 TB high-end servers are a lot closer than you think.
A lot closer - though ours is really 10TB online, with 300+ TB nearline.
Note - most these really large filesystems allow the inode tables and bitmaps to be stored on disks with a relatively small blocksize (raid 5), and the data on different drives (striped) with a large block size (I believe ours is 64K to 128K sized data blocks, inode/bitmaps are 16K-32K.) This is done for two reasons:
1. the data blocks are temporary until the the file migrates to tape and is striped. If we loose a disk out of the stripe, the filesystem is sort of dead. Replace the disk and all is well since the metadata IS protected. Files that were physically damanged are recalled from tape. New files not yet migrated are destroyed, but since the tape archive is created as near the file creation date (close time), it doesn't really happen that often. Not many files will be damaged.
2. The inode/bitmap metadata is raid5 for maximum protection. Backup of just the metadata can also be done much faster (about 3 hours in our case).
The division allows for high integrity of the meta data (which is also backed up daily (incremental) - but without the corresponding datablocks), along with maximum capacity for data. 1/5 of 200TB is about 40TB if raid5 were used with everything.
And for the curious, the filesystems are SAMFS and SAMQFS on Sun E10000s. We migrated the data from Cray NC1 filesystems with DMF - Cray data migration facility (this took over 4 months. Would have taken only a month or two, but we also had to accept new data at the same time).
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Quoted 4 lines · Jesse Pollard
And for the curious, the filesystems are SAMFS and SAMQFS on Sun E10000s. We migrated the data from Cray NC1 filesystems with DMF - Cray data migration facility (this took over 4 months. Would have taken only a month or two, but we also had to accept new data at the same time).
Thanks for the fascinating data, however you left out one crucial piece of
information: how many data bits in your processor?
Signature · 1 line
Daniel
--------- Received message begins Here ---------
Quoted 9 lines · Jesse Pollard
Quoted 4 lines
And for the curious, the filesystems are SAMFS and SAMQFS on Sun E10000s. We migrated the data from Cray NC1 filesystems with DMF - Cray data migration facility (this took over 4 months. Would have taken only a month or two, but we also had to accept new data at the same time).
Thanks for the fascinating data, however you left out one crucial piece of
information: how many data bits in your processor?
Sorry - I did omit that..
Sun E10000s are Sparc based, and 64 bit. I realize this doesn't refer to the 32 bit boundaries, but they were operating in 32 bit mode for the initial installation (Solaris 2.6) and since some of the filesystem support was 3rd party and didn't work when the kernel was in 64 bit mode.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Quoted 5 lines · Jesse Pollard
Note - most these really large filesystems allow the inode tables and bitmaps to be stored on disks with a relatively small blocksize (raid 5), and the data on different drives (striped) with a large block size (I believe ours is 64K to 128K sized data blocks, inode/bitmaps are 16K-32K.) This is done for two reasons:
Since we're on this subject, and you have experience with these large block sizes, where exactly do you see the large savings?
- setup cost of the disk transfer?
- rotational latency of small transfers?
- setup cost of the network transfer?
- interrupt processing overhead?
- other?
Signature · 1 line
Daniel
Quoted 8 lines · Daniel Phillips
Quoted 5 lines · Jesse Pollard
Note - most these really large filesystems allow the inode tables and bitmaps to be stored on disks with a relatively small blocksize (raid 5), and the data on different drives (striped) with a large block size (I believe ours is 64K to 128K sized data blocks, inode/bitmaps are 16K-32K.) This is done for two reasons:
Quoted 8 lines
Since we're on this subject, and you have experience with these large block sizes, where exactly do you see the large savings?
- setup cost of the disk transfer?
- rotational latency of small transfers?
- setup cost of the network transfer?
- interrupt processing overhead?
- other?
If you surf on over to
http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.15/ you'll see some code which performs 64k I/Os. Reads direct into pagecache. It reduces the cost of reading from disk by 25% in my testing. (That code is ready to go - just waiting for Linus to rematerialise).
The remaining profile is interesting. The workload is simply `cat large_file > /dev/null':
c012b448 33 0.200877 kmem_cache_free c0131af8 33 0.200877 flush_all_zero_pkmaps c01e51bc 33 0.200877 blk_recount_segments c01f9aec 34 0.206964 hpt374_udma_stop c016eb80 36 0.219138 ext2_get_block c0133320 37 0.225225 page_cache_readahead c013740c 37 0.225225 __getblk c0131ba0 41 0.249574 kmap_high c01fa1c4 41 0.249574 ata_start_dma c016e7dc 46 0.28001 ext2_block_to_path c01e5320 48 0.292184 blk_rq_map_sg c01c65d0 50 0.304358 radix_tree_reserve c014bfb0 53 0.32262 do_mpage_bio_readpage c01f4d88 54 0.328707 ata_irq_request c0136b34 64 0.389579 __get_hash_table c0126a00 72 0.438276 do_generic_file_read c016e910 82 0.499148 ext2_get_branch c0126610 88 0.535671 unlock_page c0106df4 91 0.553932 system_call c012b04c 94 0.572194 kmem_cache_alloc c01f2494 126 0.766983 ata_taskfile c01c66e8 163 0.992208 radix_tree_lookup c012d250 165 1.00438 rmqueue c0105274 2781 16.9284 default_idle c0126e48 11009 67.0136 file_read_actor
That's a single 500MHz PIII Xeon, reading at 35 megabytes/sec.
There's 17% "overhead" here. Going to a larger filesystem blocksize would provide almost zero benefit in the I/O layers.
Savings from larger blocks and larger pages would come into the radix tree operations, get_block, a few other places. At a guess, 8k blocks would cut the overhead to 10-12%.
And larger block size significantly penalises bandwidth for the many-small-file case. The larger the blocks, the worse it gets. You end up having to implement complexities such as tail-merging to get around the inefficiency which the workaround for your other inefficiency caused.
And larger pages with small blocks isn't an answer - CPU load and seek costs from 2-blocks-per-page is measurable. At 4 blocks-per-page it's getting serious.
Small pages and pagesize=blocksize are good. I see no point in going to larger pages or blocks until the current scheme is working efficiently and has been *proven* to still be unfixably inadequate.
The current code sucks. Simply amortising that suckiness across larger blocks is not the right thing to do.
-
Quoted 9 lines · Jesse Pollard
Note - most these really large filesystems allow the inode tables and bitmaps to be stored on disks with a relatively small blocksize (raid 5), and the data on different drives (striped) with a large block size (I believe ours is 64K to 128K sized data blocks, inode/bitmaps are 16K-32K.)
The division allows for high integrity of the meta data (which is also backed up daily (incremental) - but without the corresponding datablocks), along with maximum capacity for data. 1/5 of 200TB is about 40TB if raid5 were used with everything.
Interestingly, this can also be done for Lustre, the OSS cluster filesystem I am currently working on (http://www.clusterfs.com).
All of the filesystem metadata is actually stored on a separate server, so its disk can be configured totally separately from the file data.
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"DP" == Daniel Phillips <[email protected]> writes:
DP> Incidently, the 200 TB high-end servers are a lot closer than you DP> think.
Perhaps the "low-end server" is a more interesting case, though. After all, you can put over 4TB in one machine now for somewhere near $12K. (32 160GB IDE disks on four 3ware 7850 cards running RAID5, all striped into one huge "RAID50" array.) This is only going to get cheaper, and when someone out-does Maxtor for the "big IDE disk" crown, the capacity will jump again. Of course it doesn't have the same performance or extreme reliability of the more expensive solutions, but hey, it only costs $12K.
Still, 16TB done this way looks to be a few years away yet.
- J<