Skip to contentSkip to conversation

4K drives, sectsz=512, bsize=4096

DISCUSSION Michael Monnerie 5 participants 12 messages Aug 30, 2010 – Sep 1, 2010

lore
Share & tools
lore
#0 lore raw

I have 2x 2TB 4K sector drives, joined with LVM to RAID-0. On top of that, I created an XFS, but that has sectsz=512.

Will there be any real difference if I re-format with sectsz=4096? AFAIK, XFS will do I/O based on block size, so the sector size doesn't do any harm. Is that correct?

A question for LVM: Is there anything I need to tell to LVM to let it know that those are 4K sector drives and I/O should be aligned to that? Drives are reported as 512b sectors, but really are 4K. There seems to be no way to instruct the kernel to see those drives as 4K drives.

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 3 lines · On Montag, 30. August 2010 Michael Monnerie wrote:

Will there be any real difference if I re-format with sectsz=4096? AFAIK, XFS will do I/O based on block size, so the sector size
doesn't do any harm. Is that correct?

Nobody? Is my assumption correct that a change in sectsz from 512 to 4096 would be of no difference for the filesystem?

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 8 lines · Michael Monnerie
Quoted 3 lines · On Montag, 30. August 2010 Michael Monnerie wrote:

Will there be any real difference if I re-format with sectsz=4096? AFAIK, XFS will do I/O based on block size, so the sector size
doesn't do any harm. Is that correct?

Nobody? Is my assumption correct that a change in sectsz from 512 to 4096 would be of no difference for the filesystem?

Just be sure your first sector of your top-level block device is 4k aligned, and then yes you'll want to set sector size to 4k so that xfs will do all log IO in 4k blocks as well.

If you do it right (and especially vs. if you do it wrong) it should be a bit faster if all IOs are 4k aligned on the disk.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 3 lines · On Dienstag, 31. August 2010 Eric Sandeen wrote:

Just be sure your first sector of your top-level block device is 4k aligned, and then yes you'll want to set sector size to 4k so that xfs will do all log IO in 4k blocks as well.

I can only hope LVM does it right, I have no idea how it aligns.

Quoted 2 lines

If you do it right (and especially vs. if you do it wrong) it should be a bit faster if all IOs are 4k aligned on the disk.

And that's what's interesting me: why? Won't XFS do all I/Os at minimum for a given block size? Or is it possible XFS does write only a single sector? I'd expect the smallest I/O size to be the block size, but it seems I'm wrong?

I guess there's no way to "convert" an existing XFS with sectsz=512,bsize=4096 to sectsz=4096,bsize=4096? Maybe that's only a flag that can be changed?

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 7 lines · Michael Monnerie
Quoted 3 lines · On Dienstag, 31. August 2010 Eric Sandeen wrote:

Just be sure your first sector of your top-level block device is 4k aligned, and then yes you'll want to set sector size to 4k so that xfs will do all log IO in 4k blocks as well.

I can only hope LVM does it right, I have no idea how it aligns.

Quoted 7 lines
Quoted 2 lines

If you do it right (and especially vs. if you do it wrong) it should be a bit faster if all IOs are 4k aligned on the disk.

And that's what's interesting me: why? Won't XFS do all I/Os at minimum for a given block size? Or is it possible XFS does write only a single sector? I'd expect the smallest I/O size to be the block size, but it seems I'm wrong?

the log can do things on sector size boundaries.

Quoted 3 lines

I guess there's no way to "convert" an existing XFS with sectsz=512,bsize=4096 to sectsz=4096,bsize=4096? Maybe that's only a flag that can be changed?

I don't think so ...

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
In reply to Michael Monnerie
Quoted 9 lines · ----- "Michael Monnerie
Quoted 2 lines · On Dienstag, 31. August 2010 Eric Sandeen wrote:

If you do it right (and especially vs. if you do it wrong) it should be a bit faster if all IOs are 4k aligned on the disk.

And that's what's interesting me: why? Won't XFS do all I/Os at minimum
for a given block size? Or is it possible XFS does write only a single sector? I'd expect the smallest I/O size to be the block size, but it seems I'm wrong?

Log I/O and direct writes are sector sized & aligned.

Quoted 3 lines

I guess there's no way to "convert" an existing XFS with sectsz=512,bsize=4096 to sectsz=4096,bsize=4096? Maybe that's only a flag that can be changed?

There's no way (other than dump, mkfs & restore), the filesystem is laid out differently (most data structures, like superblocks and other metadata become 4K aligned).

cheers.

Signature · 6 lines
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
In reply to Eric Sandeen
Quoted 1 line · On Mittwoch, 1. September 2010 Eric Sandeen wrote:

the log can do things on sector size boundaries.

Great, thanks to both of you for clarification. Going to reformat now...

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
In reply to Nathan Scott

Hi,

I would like to know how the inode size should be configured for 4 kB sector drives?

Should we leave it at the default 256 bytes, or set it to the maximum of 2 kB?

Do inodes occupy discrete sectors, or do they occupy part of the filesystem block?

GL

Quoted 32 lines · Nathan Scott

From: Nathan Scott <[email protected]>
Subject: Re: 4K drives, sectsz=512, bsize=4096
To: "Michael Monnerie" <[email protected]>
Cc: [email protected]
Date: Wednesday, 1 September, 2010, 6:34 AM

----- "Michael Monnerie" <[email protected]>

Quoted 2 lines · wrote:
Quoted 1 line · On Dienstag, 31. August 2010 Eric Sandeen wrote:

If you do it right (and especially vs. if you do

it wrong) it

Quoted 1 line
Quoted 1 line

should be a bit faster if all IOs are 4k aligned

on the disk.

Quoted 2 lines


And that's what's interesting me: why? Won't XFS do

all I/Os at

Quoted 2 lines

minimum
for a given block size? Or is it possible XFS does

write only a single

Quoted 1 line

sector? I'd expect the smallest I/O size to be the

block size, but it

Quoted 1 line

seems I'm wrong?

Log I/O and direct writes are sector sized & aligned.

Quoted 1 line

I guess there's no way to "convert" an existing XFS

with

Quoted 1 line

sectsz=512,bsize=4096 to sectsz=4096,bsize=4096? Maybe

that's only a

Quoted 1 line

flag that can be changed?

Quoted 16 lines

There's no way (other than dump, mkfs & restore), the filesystem is laid
out differently (most data structures, like superblocks and other metadata
become 4K aligned).

cheers.

Signature · 6 lines
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
Quoted 4 lines · ----- "Gim Leong Chin

Hi,

I would like to know how the inode size should be configured for 4 kB sector drives?

There is no direct correlation between inode size and sector size.

Quoted 2 lines

Should we leave it at the default 256 bytes, or set it to the maximum of 2 kB?

See above. Keep the defaults unless you have a good reason not to.

Quoted 2 lines

Do inodes occupy discrete sectors, or do they occupy part of the filesystem block?

They occupy part of a filesystem block.

cheers.

Signature · 6 lines
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw
In reply to Gim Leong Chin
Quoted 2 lines · On Mittwoch, 1. September 2010 Gim Leong Chin wrote:

Should we leave it at the default 256 bytes, or set it to the maximum
of 2 kB?

An interesting question. Why are inodes sizes configurable at all? To store ACLs? How would one know when bigger Inodes should be used? And what is the implication when they would be needed but aren't used?

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw
In reply to Michael Monnerie
Quoted 6 lines · Michael Monnerie

I have 2x 2TB 4K sector drives, joined with LVM to RAID-0. On top of that, I created an XFS, but that has sectsz=512.

Will there be any real difference if I re-format with sectsz=4096? AFAIK, XFS will do I/O based on block size, so the sector size doesn't do any harm. Is that correct?

As nathan mentioned XFS issues log I/O based on the sector size, and we do allow direct I/O down to the sector size. That's two reasons why or why you don't want a 4096 byte sector size. Depending on the rmv implementation of the drive log I/O might be really slow if you set the 512 byte sector size. On the other hand there's lots of applications that have the 512 byte alignment for direct I/O hardcoded, which will break if you have a larger sector size. That's one of the fun things 4k sector size disks will bring up, especially the real 4k SAS disks that do not accept any I/O smaller than that.

Quoted 4 lines

A question for LVM: Is there anything I need to tell to LVM to let it know that those are 4K sector drives and I/O should be aligned to that? Drives are reported as 512b sectors, but really are 4K. There seems to be no way to instruct the kernel to see those drives as 4K drives.

If you have a recent enough LVM all metdata is aligned on 1MB boundaries which will do just fine.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
In reply to Michael Monnerie
Quoted 7 lines · Michael Monnerie
Quoted 2 lines · On Mittwoch, 1. September 2010 Gim Leong Chin wrote:

Should we leave it at the default 256 bytes, or set it to the maximum
of 2 kB?

An interesting question. Why are inodes sizes configurable at all? To store ACLs? How would one know when bigger Inodes should be used? And what is the implication when they would be needed but aren't used?

The XFS inode consists of three parts:

- the fixed format dinode
- the data fork
- the attribute fork

the fixed format inode is fixed size, so any change in the inode size only applies to the data and attribute forks. For regular files we generally don't use much space in the data fork as it just contains the extent list, and most files have rather few of them. But we can also store short smbolic links directly inside it, as well as the content of directories. The attribute fork is used to store extent attributes and if it's large enough we can store them inline instead of using external blocks.

You want large inodes mostly if you store lots of extentded attributes, either for ACLs, Selinux or posisbly DMAPI. It will also help if you have enough directories that are just too big for the inline directory format with smaller inode sizes.

Quoted 22 lines
Signature · 21 lines
-- 
mit freundlichen Gr?ssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei H?user zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/



_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

---end quoted text---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs