Recommendation on XFS Filesystem Creation, Lefthand P4500, iSCSI ?

Hello List.

We recently purchased HP Lefthand 4500 G2 24TB iSCSI SAN Box. The Box is configured in RAID6. (and yes, we are going to expand that with another boxes in near future).

I'm running Centos 6.2 with the latest updates on the iSCSI initiator machine. HP DL380 G7 with 24GB in RAM

I need to create a 7TB volume at the start, and grow it in near future.

Are there any recommendation on creating that XFS volume ? Particularly regarding iSCSI and that big ?

This server will act as a fileserver for 50+ machines serving request with netatalk and samba.

Surely I will need to use inode64 option. Right ?

Any recommendation would be much appreciated.

Thanks allot people.
Best regards,

Svavar
Reykjavik - Iceland

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi Svavar,

We have a couple of NAS servers (just direct disk connection, not iSCSI) with XFS filesystems up to ~30TB, but didn't do anything special for those (just defaults on everything, iirc).

The only problem we had was with filesystems bigger than that, where we get kernel crashes due to XFS overflowing the stack. Apparently this may be worse if you NFS-share XFS layered over LVM (more stuff on the stack, I guess), but I imagine you'd be fine with 7TB.

Cheers,

Mike.

<br />
<hr />
<p><font face="Arial" size="1"> Plymouth Marine Laboratory<br /> Registered Office: <br />
Prospect Place<br />
The Hoe<br />
Plymouth PL1 3DH
</font></p>

<p><font face="Arial" size="1">Website: <a href="http://www.pml.ac.uk">www.pml.ac.uk</a>
<br />
<a href="http://www.pml.ac.uk/pdf/PML%20Annual%20Review%202011_2.pdf">Click here for the latest PML Annual Review</a> <br />
Registered Charity No. 1091222<br />
PML is a company limited by guarantee<br />
registered in England & Wales<br />
company number 4178503</font></p>

<p><font face="Arial" size="1" color="green">Please think before you print.</font></p>

<hr />

<p><font face="Arial" size="1">This e-mail, its content and any file attachments are confidential.</font></p>

<p><font face="Arial" size="1">If you have received this e-mail in error please do not copy, disclose it to any third party or use the contents or attachments in any way. Please notify the sender by replying to this e-mail or e-mail [email protected] and then delete the email without making any copies or using it in any other way.</font></p>

<p><font face="Arial" size="1">The content of this message may contain personal views which are not the views of Plymouth Marine Laboratory unless specifically stated.</font></p>

<p><font face="Arial" size="1">You are reminded that e-mail communications are not secure and may contain viruses. Plymouth Marine Laboratory accepts no liability for any loss or damage which may be caused by viruses.</font></p>

<hr />
<br />
<br />

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Why start small and grow? What's the point? Usually "grow" has no problems, but any restructuring operation (on RAID or filetrees) is a dangerous moment. Begging for trouble.

Ideally create multiple small volumes over time and assign different sets of users to them. For better 'fsck' and safety if anything.

Hopefully you won't be creating a single 24TB RAID6 set, but that kind of (euphemism) strategy seems to be standard.

I would not configure the volumes exported by the iSCSI server as sparse/thin, but in that case using 'discard' should help.

As an obvious note, but just to be sure, you will be running a 64b OS. But still be careful about stack overflows. You are using 'netatalk' and Samba which are user-level services, so those won't impact the kernel stack, but DM/LVM and NFS would. A pity as the only good use of DM would be to create a snapshot volume for "frozen" state backups. But it should still work, after all 64b kernels have a larger kernel stack frame.

Then the major issue: backups. How are you going to create at least 3-4 backups copies of your filetree(s)? Interesting issue.

If you have performance requirements ideally the server will be dual homed on an iSCSI-only *physical* (not VLAN) network and a distinct user network (with different NICs). Tuning the Linux IP/TCP parameters might help a fair bit too, and a 10Gb network would be a big improvement over a 1Gb one, and short range 10Gb cards are quite affordable (I particularly like Myri cards, but the Dell Broadcom ones are also good).

Yes (IIRC it will soon become the default), as recommended in
http://www.spinics.net/lists/xfs/msg11455.html; as it says, 'inode64' has a secondary effect on allocation, which usually helps, and in your case I think it does; anyhow with filetrees much above 1TiB (with 512B sectors) it is hard to avoid 'inode64'.

But perhaps other most important point is to ensure that 'su=' and 'sw=' are set appropriately as you are using parity RAID. Usually the 'mkfs.xfs' asks the Linux kernel for geometry details, which usually works when the RAID set is provided by MD or DM, but in your case probably the RAID geometry will not be available from the iSCSI layer, so probably you need to specify it manually.

Consider whether to enable barriers, and the answer is probably yes (if supported), unless you have a really reliable storage chains with battery backups and a very reliable kernel too.

Dramatically reducing the number of "dirty" pages in the page cache usually is rather important too. It might be of benefit to set the elevator to "noop".

Probably a sector size of 4096 is also a good idea, and on recent storage system this is always true where possible. Having jumbo frames enabled on your network (the iSCSI facing one) would probably help a fair bit, regardless of sector size.

If you use lots of ACLs (common for fileservers) consider having a larger-than-usual inode size, which also helps if you expect to have large extent maps (very huge files or very fragmented ones, for example mail archives). I personally tend to have 2048B inodes, but that is an arguable choice.

If you have a very reliable storage chain and high rates of file creation and deletion you may want an external journal, but probably your iSCSI host adapter has enough BBU backed cache that it is not necessary. You may want to disable delayed logging to reduce the vulnerability window otherwise.

PS I have adeptad here a number of hints about setting up XFS
(and other types) of filetrees taken from:
http://www.sabi.co.uk/Notes/linuxFS.html#fsHintsXFS

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs