makefs alignment issue

Just remade a couple of filesystems and received an alignment msg I don't recall receiving previously:

# mkfs.xfs -f -d su=64k,sw=12 /dev/s2d_a1l003
mkfs.xfs: Specified data stripe width 1536 is not the same as the volume
stripe width 2048

meta-data=/dev/s2d_a1l003 isize=256 agcount=44,
agsize=268435440 blks

         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=11709285376, imaxpct=5
         =                       sunit=16     swidth=192 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

/dev/s2d_a1l003 is an alias to dm-0 which is a dm-multipath device to a LUN on hardware RAID. The hardware geometry is 64kx 12, 768k. AFAIK no geometry information has been specified for these device mapper devices (someone else's responsibility). Though I assume dm geometry data is the reason for mkfs.xfs throwing this alert. I don't find anything in /sys/devices/virtual/block/dm-0/ indicating geometry.

Any ideas how to verify what's going on here and fix it?

Quoted 6 lines · Stan Hoeppner

Just remade a couple of filesystems and received an alignment msg I don't recall receiving previously:

# mkfs.xfs -f -d su=64k,sw=12 /dev/s2d_a1l003
mkfs.xfs: Specified data stripe width 1536 is not the same as the volume
stripe width 2048

so specified geometry that differs from the underlying device...

Quoted 11 lines

meta-data=/dev/s2d_a1l003 isize=256 agcount=44,
agsize=268435440 blks

         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=11709285376, imaxpct=5
         =                       sunit=16     swidth=192 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Quoted 8 lines

/dev/s2d_a1l003 is an alias to dm-0 which is a dm-multipath device to a LUN on hardware RAID. The hardware geometry is 64kx 12, 768k. AFAIK no geometry information has been specified for these device mapper devices (someone else's responsibility). Though I assume dm geometry data is the reason for mkfs.xfs throwing this alert. I don't find anything in /sys/devices/virtual/block/dm-0/ indicating geometry.

Any ideas how to verify what's going on here and fix it?

# blockdev --getiomin --getioopt /dev/s2d_a1l003

The first number, minimum io size, is what is used for sunit The 2nd number, optimal io size, is what is used for swidth

Where dm got the geometry, I'm not sure - you'd have to look into how you set up the dm device, and what its defaults are I think.

On 10/24/2014 03:14 PM, Eric Sandeen wrote:

Quoted 9 lines

Quoted 1 line

Any ideas how to verify what's going on here and fix it?

# blockdev --getiomin --getioopt /dev/s2d_a1l003

The first number, minimum io size, is what is used for sunit The 2nd number, optimal io size, is what is used for swidth

Where dm got the geometry, I'm not sure - you'd have to look into how you set up the dm device, and what its defaults are I think.

Looks like they're being passed up the stack:

# blockdev --getiomin --getioopt /dev/dm-0
512
1048576

# multipath -ll
3600c0ff0003630917954075401000000 dm-0 Tek,DH6554
size=44T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 9:0:0:3 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:0:3 sdf 8:80  active ready running

/sys/block/sdj/queue# cat minimum_io_size
512
/sys/block/sdj/queue# cat optimal_io_size
1048576

root@Anguish-ssu-1:/sys/block/sdf/queue# cat minimum_io_size
512
root@Anguish-ssu-1:/sys/block/sdf/queue# cat optimal_io_size
1048576

So it's the presence of a value in optimal_io_size that causes the problem here. My single disk workstation has min but not optimal, as I assume most do. And I don't get this msg when formatting it.

It's interesting that mkfs.xfs would use these values given they are static firmware values in most controllers. Thus they don't change when one uses different geometry for different arrays/LUNs...

So it seems I can safely ignore this mkfs msg.

Thanks,
Stan

Quoted 2 lines · Stan Hoeppner

On 10/24/2014 03:14 PM, Eric Sandeen wrote:

...

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Any ideas how to verify what's going on here and fix it?

# blockdev --getiomin --getioopt /dev/s2d_a1l003

Quoted 4 lines · Eric Sandeen

Quoted 2 lines · Stan Hoeppner

On 10/24/2014 03:14 PM, Eric Sandeen wrote:

Quoted 5 lines

...

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Any ideas how to verify what's going on here and fix it?

# blockdev --getiomin --getioopt /dev/s2d_a1l003

Also, what does it show for the underlying non-multipath device(s)?

Quoted 11 lines · Eric Sandeen

Quoted 9 lines · Eric Sandeen

Quoted 2 lines · Stan Hoeppner

On 10/24/2014 03:14 PM, Eric Sandeen wrote:

...

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Any ideas how to verify what's going on here and fix it?

# blockdev --getiomin --getioopt /dev/s2d_a1l003

Quoted 1 line

Also, what does it show for the underlying non-multipath device(s)?

# blockdev --getiomin --getioopt /dev/sdj
512
1048576
# blockdev --getiomin --getioopt /dev/sdf
512
1048576

Quoted 12 lines · Stan Hoeppner

Quoted 11 lines · Eric Sandeen

Quoted 9 lines · Eric Sandeen

Quoted 2 lines · Stan Hoeppner

On 10/24/2014 03:14 PM, Eric Sandeen wrote:

...

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Any ideas how to verify what's going on here and fix it?

# blockdev --getiomin --getioopt /dev/s2d_a1l003

Quoted 8 lines

Quoted 1 line

Also, what does it show for the underlying non-multipath device(s)?

# blockdev --getiomin --getioopt /dev/sdj
512
1048576
# blockdev --getiomin --getioopt /dev/sdf
512
1048576

Ok, so dm multipath is just bubbling up what the device itself is claiming; not dm's doing.

I forgot to ask (and you forgot to report...!) what version of xfsprogs you're using....

Currently, blkid_get_topology() in xfsprogs does:

/*
* Blkid reports the information in terms of bytes, but we want it in
* terms of 512 bytes blocks (just to convert it to bytes later..) *
* If the reported values are the same as the physical sector size
* do not bother to report anything. It will just cause warnings
* if people specify larger stripe units or widths manually. */
val = blkid_topology_get_minimum_io_size(tp);
if (val > *psectorsize)
*sunit = val >> 9;
val = blkid_topology_get_optimal_io_size(tp);
if (val > *psectorsize)
*swidth = val >> 9;

so in your case sunit probably wouldn't get set (can you confirm with

# blockdev --getpbsz that the physical sector size is also 512?)

But the optimal size is > physical sector so swidth gets set.

Bleah... can you just collect all of:

# blockdev --getpbsz --getss --getiomin --getioopt

for your underlying devices, and I'll dig into how xfsprogs is behaving for those values. I have a hunch that we should be ignoring stripe units of 512 even if the "width" claims to be something larger.

If the same interface is used for Linux logical block devices (md, dm, lvm, etc) and hardware RAID, I have a hunch it may be better to determine that, if possible, before doing anything with these values. As you said previously, and I agree 100%, a lot of RAID vendors don't

export meaningful information here. In this specific case, I think the RAID engineers are exporting a value, 1 MB, that works best for their cache management, or some other path in their firmware. They're concerned with host interface xfer into the controller, not the IOs on the back end to the disks. They don't see this as an end-to-end deal. In fact, I'd guess most of these folks see their device as performing magic, and it doesn't matter what comes in or goes out either end. "We'll take care of it."

optimal_io_size. I'm guessing this has different meaning for different folks. You say optimal_io_size is the same as RAID width. Apply that to this case:

hardware RAID 60 LUN, 4 arrays
16+2 RAID6, 256 KB stripe unit, 4096 KB stripe width 16 MB LUN stripe width
optimal_io_size = 16 MB

Is that an appropriate value for optimal_io_size even if this is the RAID width? I'm not saying it isn't. I don't know. I don't know what other layers of the Linux and RAID firmware stacks are affected by this, nor how they're affected.

Quoted 4 lines · Stan Hoeppner

If the same interface is used for Linux logical block devices (md, dm, lvm, etc) and hardware RAID, I have a hunch it may be better to determine that, if possible, before doing anything with these values. As you said previously, and I agree 100%, a lot of RAID vendors don't

Quoted 8 lines

export meaningful information here. In this specific case, I think the RAID engineers are exporting a value, 1 MB, that works best for their cache management, or some other path in their firmware. They're concerned with host interface xfer into the controller, not the IOs on the back end to the disks. They don't see this as an end-to-end deal. In fact, I'd guess most of these folks see their device as performing magic, and it doesn't matter what comes in or goes out either end. "We'll take care of it."

Deja vu. This is an isochronous RAID array you are having trouble with, isn't it?

FWIW, do your problems go away when you make you hardware LUN width a multiple of the cache segment size?

Quoted 9 lines

optimal_io_size. I'm guessing this has different meaning for different folks. You say optimal_io_size is the same as RAID width. Apply that to this case:

hardware RAID 60 LUN, 4 arrays
16+2 RAID6, 256 KB stripe unit, 4096 KB stripe width 16 MB LUN stripe width
optimal_io_size = 16 MB

Quoted 4 lines

Is that an appropriate value for optimal_io_size even if this is the RAID width? I'm not saying it isn't. I don't know. I don't know what other layers of the Linux and RAID firmware stacks are affected by this, nor how they're affected.

yup, i'd expect minimum = 4MB (i.e stripe unit 4MB so we align to the underlying RAID6 luns) and optimal = 16MB for the stripe width (and so with swalloc we align to the first lun in the RAID0).

This should be passed up unchanged through the stack if none of the software layers are doing other geometry modifications (e.g. more raid, thinp, etc).

Quoted 14 lines · Dave Chinner

Quoted 12 lines · Stan Hoeppner

If the same interface is used for Linux logical block devices (md, dm, lvm, etc) and hardware RAID, I have a hunch it may be better to determine that, if possible, before doing anything with these values. As you said previously, and I agree 100%, a lot of RAID vendors don't export meaningful information here. In this specific case, I think the RAID engineers are exporting a value, 1 MB, that works best for their cache management, or some other path in their firmware. They're concerned with host interface xfer into the controller, not the IOs on the back end to the disks. They don't see this as an end-to-end deal. In fact, I'd guess most of these folks see their device as performing magic, and it doesn't matter what comes in or goes out either end. "We'll take care of it."

Quoted 2 lines

Deja vu. This is an isochronous RAID array you are having trouble with, isn't it?

I don't believe so. I'm pretty sure the parity rotates; i.e. standard RAID5/6.

Quoted 2 lines

FWIW, do your problems go away when you make you hardware LUN width a multiple of the cache segment size?

Hadn't tried it. And I don't have the opportunity now as my contract has ended. However the problems we were having weren't related to controller issues but excessive seeking. I mentioned this in that (rather lengthy) previous reply.

Quoted 9 lines

Quoted 9 lines

optimal_io_size. I'm guessing this has different meaning for different folks. You say optimal_io_size is the same as RAID width. Apply that to this case:

hardware RAID 60 LUN, 4 arrays
16+2 RAID6, 256 KB stripe unit, 4096 KB stripe width 16 MB LUN stripe width
optimal_io_size = 16 MB

Quoted 8 lines

Quoted 4 lines

Is that an appropriate value for optimal_io_size even if this is the RAID width? I'm not saying it isn't. I don't know. I don't know what other layers of the Linux and RAID firmware stacks are affected by this, nor how they're affected.

yup, i'd expect minimum = 4MB (i.e stripe unit 4MB so we align to the underlying RAID6 luns) and optimal = 16MB for the stripe width (and so with swalloc we align to the first lun in the RAID0).

At minimum 4MB how does that affect journal writes which will be much smaller, especially with a large file streaming workload, for which this setup is appropriate? Isn't the minimum a hard setting? I.e. we can never do an IO less than 4MB? Do other layers of the stack use this variable? Are they expecting values this large?

Quoted 18 lines · Stan Hoeppner

Quoted 16 lines · Dave Chinner

Quoted 4 lines · Stan Hoeppner

If the same interface is used for Linux logical block devices (md, dm, lvm, etc) and hardware RAID, I have a hunch it may be better to determine that, if possible, before doing anything with these values. As you said previously, and I agree 100%, a lot of RAID vendors don't

Quoted 8 lines

export meaningful information here. In this specific case, I think the RAID engineers are exporting a value, 1 MB, that works best for their cache management, or some other path in their firmware. They're concerned with host interface xfer into the controller, not the IOs on the back end to the disks. They don't see this as an end-to-end deal. In fact, I'd guess most of these folks see their device as performing magic, and it doesn't matter what comes in or goes out either end. "We'll take care of it."

Deja vu. This is an isochronous RAID array you are having trouble with, isn't it?

Quoted 2 lines

I don't believe so. I'm pretty sure the parity rotates; i.e. standard RAID5/6.

The location of parity doesn't dtermine that it is isochronous in behaviour or not. Often RAID5/6 is marketing speak for "single/dual parity", not the type of redundancy that is implemented in the hardware ;)

Quoted 7 lines

Quoted 2 lines

FWIW, do your problems go away when you make you hardware LUN width a multiple of the cache segment size?

Hadn't tried it. And I don't have the opportunity now as my contract has ended. However the problems we were having weren't related to controller issues but excessive seeking. I mentioned this in that (rather lengthy) previous reply.

Right, but if you had a 768k stripe width and a 1MB cache segment size, a cache segment operation would require two stripe widths to be operated on, and only one would be a whole stripe width. hence the possibility of doing more IOs than are necessary to populate or write back cache segments. i.e. it's a potential reason for why the back end disks didn't have anywhere near the expected seek capability they were supposed to have....

Quoted 18 lines

Quoted 9 lines

Quoted 9 lines

optimal_io_size. I'm guessing this has different meaning for different folks. You say optimal_io_size is the same as RAID width. Apply that to this case:

hardware RAID 60 LUN, 4 arrays
16+2 RAID6, 256 KB stripe unit, 4096 KB stripe width 16 MB LUN stripe width
optimal_io_size = 16 MB

Quoted 8 lines

Quoted 4 lines

Is that an appropriate value for optimal_io_size even if this is the RAID width? I'm not saying it isn't. I don't know. I don't know what other layers of the Linux and RAID firmware stacks are affected by this, nor how they're affected.

yup, i'd expect minimum = 4MB (i.e stripe unit 4MB so we align to the underlying RAID6 luns) and optimal = 16MB for the stripe width (and so with swalloc we align to the first lun in the RAID0).

Quoted 5 lines

At minimum 4MB how does that affect journal writes which will be much smaller, especially with a large file streaming workload, for which this setup is appropriate? Isn't the minimum a hard setting? I.e. we can never do an IO less than 4MB? Do other layers of the stack use this variable? Are they expecting values this large?

No, "minimum_io_size" is for "minimum *efficient* IO size" not the smallest supported IO size. The smallest supported IO sizes and atomic IO sizes are defined by hw_sector_size, physical_block_size and logical_block_size.