Skip to contentSkip to conversation

DAX 2MB mappings for XFS

DISCUSSION Kani, Toshi 3 participants 7 messages Jan 12, 2018 – Jan 13, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

I noticed that DAX 2MB mmap no longer works on XFS. I used the following steps on a 4.15-rc7 kernel. Am I missing something, or is there a problem in XFS?

# mkfs.xfs -f -d su=2m,sw=1 /dev/pmem0
# mount -o dax /dev/pmem0 /mnt/pmem0
# xfs_io -c "extsize 2m" /mnt/pmem0

fio with libpmem engine (which uses mmap) is slow since it gets serialized by 4KB page faults.

# numactl --cpunodebind=0 --membind=0 fio --filename=/mnt/pmem0/testfile 
--rw=read --ioengine=libpmem --iodepth=1 --numjobs=16 --runtime=60 --
group_reporting --name=perf_test --thread=1 --size=6g --bs=128k --
direct=1
  :
Run status group 0 (all jobs):
   READ: bw=4357MiB/s (4569MB/s), 4357MiB/s-4357MiB/s (4569MB/s-
4569MB/s), io=96.0GiB (103GB), run=22560-22560msec

Resulted file blocks in "testfile" are not aligned by 2MB.

# filefrag -v /mnt/pmem0/testfile
Filesystem type is: 58465342
File size of testfile is 6442450944 (1572864 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected:
flags:
   0:        0..  261111:        520..    261631: 261112:
   1:   261112..  261348:         12..       248:    237:     261632:
   2:   261349..  522705:     261644..    523000: 261357:        249:
   3:   522706..  784062:     523276..    784632: 261357:     523001:
   4:   784063.. 1045419:     784908..   1046264: 261357:     784633:
   5:  1045420.. 1304216:    1049100..   1307896: 258797:    1046265:
   6:  1304217.. 1565573:    1308172..   1569528: 261357:    1307897:
   7:  1565574.. 1572863:    1570304..   1577593:   7290:    1569529: 
last,eof
testfile: 8 extents found

A file created by fallocate also shows that physical offset starts from 520, which is not aligned by 2MB.

# fallocate --length 1G /mnt/pmem0/data
# filefrag -v /mnt/pmem0/data
Filesystem type is: 58465342
File size of /mnt/pmem0/data is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected:
flags:
   0:        0..  260607:        520..    261127:
260608:             unwritten
   1:   260608..  262143:     262144..    263679:   1536:     261128:
last,unwritten,eof
/mnt/pmem0/data: 2 extents found

ext4 does not have the issue in the same steps.

# mkfs.ext4 -b 4096 -E stride=512 -F /dev/pmem1
# mount -o dax /dev/pmem1 /mnt/pmem1
# numactl --cpunodebind=0 --membind=0 fio --filename=/mnt/pmem1/testfile 
--rw=read --ioengine=libpmem --iodepth=1 --numjobs=16 --runtime=60 --
group_reporting --name=perf_test --thread=1 --size=6g --bs=128k --
direct=1      
  :
Run status group 0 (all jobs):
   READ: bw=44.4GiB/s (47.7GB/s), 44.4GiB/s-44.4GiB/s (47.7GB/s-
47.7GB/s), io=96.0GiB (103GB), run=2160-2160msec

All blocks are aligned by 2MB.

# filefrag -v /ment/pmem1/testfile
Filesystem type is: ef53
File size of testfile is 6442450944 (1572864 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected:
flags:
   0:        0..   32767:      34816..     67583:  32768:
   1:    32768..   63487:      67584..     98303:  30720:
   2:    63488..   96255:     100352..    133119:  32768:      98304:
   3:    96256..  126975:     133120..    163839:  30720:
    :
# fallocate --length 1G /mnt/pmem1/data
# filefrag -v /mnt/pmem1/data
Filesystem type is: ef53
File size of /mnt/pmem1/data is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected:
flags:
   0:        0..   30719:      34816..     65535:  30720:   unwritten
   1:    30720..   61439:      65536..     96255:  30720:   unwritten
   2:    61440..   63487:      96256..     98303:   2048:   unwritten
   :

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#1 lore raw
Quoted 43 lines · Toshi

Hello,

I noticed that DAX 2MB mmap no longer works on XFS. I used the following steps on a 4.15-rc7 kernel. Am I missing something, or is there a problem in XFS?

# mkfs.xfs -f -d su=2m,sw=1 /dev/pmem0
# mount -o dax /dev/pmem0 /mnt/pmem0
# xfs_io -c "extsize 2m" /mnt/pmem0

fio with libpmem engine (which uses mmap) is slow since it gets serialized by 4KB page faults.

# numactl --cpunodebind=0 --membind=0 fio --filename=/mnt/pmem0/testfile 
--rw=read --ioengine=libpmem --iodepth=1 --numjobs=16 --runtime=60 --
group_reporting --name=perf_test --thread=1 --size=6g --bs=128k --
direct=1
  :
Run status group 0 (all jobs):
   READ: bw=4357MiB/s (4569MB/s), 4357MiB/s-4357MiB/s (4569MB/s-
4569MB/s), io=96.0GiB (103GB), run=22560-22560msec

Resulted file blocks in "testfile" are not aligned by 2MB.

# filefrag -v /mnt/pmem0/testfile
Filesystem type is: 58465342
File size of testfile is 6442450944 (1572864 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected:
flags:
   0:        0..  261111:        520..    261631: 261112:
   1:   261112..  261348:         12..       248:    237:     261632:
   2:   261349..  522705:     261644..    523000: 261357:        249:
   3:   522706..  784062:     523276..    784632: 261357:     523001:
   4:   784063.. 1045419:     784908..   1046264: 261357:     784633:
   5:  1045420.. 1304216:    1049100..   1307896: 258797:    1046265:
   6:  1304217.. 1565573:    1308172..   1569528: 261357:    1307897:
   7:  1565574.. 1572863:    1570304..   1577593:   7290:    1569529: 
last,eof
testfile: 8 extents found

A file created by fallocate also shows that physical offset starts from 520, which is not aligned by 2MB.

Quoted 11 lines
# fallocate --length 1G /mnt/pmem0/data
# filefrag -v /mnt/pmem0/data
Filesystem type is: 58465342
File size of /mnt/pmem0/data is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected:
flags:
   0:        0..  260607:        520..    261127:
260608:             unwritten
   1:   260608..  262143:     262144..    263679:   1536:     261128:
last,unwritten,eof
/mnt/pmem0/data: 2 extents found

/me really dislikes filefrag output.

$ sudo xfs_bmap -vvp /mnt/scratch/data
/mnt/scratch/data:
 EXT: FILE-OFFSET         BLOCK-RANGE      AG AG-OFFSET          TOTAL FLAGS
   0: [0..2088959]:       4160..2093119     0 (4160..2093119)  2088960 011111
   1: [2088960..2097151]: 2101248..2109439  1 (4096..12287)       8192 010000
 FLAG Values:
    0100000 Shared extent
    0010000 Unwritten preallocated extent
    0001000 Doesn't begin on stripe unit
    0000100 Doesn't end   on stripe unit
    0000010 Doesn't begin on stripe width
    0000001 Doesn't end   on stripe width

Yeah, though so. The bmap output clearly tells me that the allocation being asked for doesn't fit into a single AG, so it's trimmed to fit.

To confirm this is the issue, let's do two smaller alllocations:

$ sudo rm /mnt/scratch/data
dave@test4:~$ sudo xfs_io -f -c "falloc 0 512m" -c "falloc 512m 512m" -c stat -c "bmap -vvp" /mnt/scratch/data
fd.path = "/mnt/scratch/data"
fd.flags = non-sync,non-direct,read-write
stat.ino = 4099
stat.type = regular file
stat.size = 1073741824
stat.blocks = 2097152
fsxattr.xflags = 0x802 [-p--------e------]
fsxattr.projid = 0
fsxattr.extsize = 2097152
fsxattr.cowextsize = 0
fsxattr.nextents = 2
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
/mnt/scratch/data:
 EXT: FILE-OFFSET         BLOCK-RANGE      AG AG-OFFSET          TOTAL FLAGS
   0: [0..1048575]:       8192..1056767     0 (8192..1056767)  1048576 010000
   1: [1048576..2097151]: 2101248..3149823  1 (4096..1052671)  1048576 010000
 FLAG Values:
    0100000 Shared extent
    0010000 Unwritten preallocated extent
    0001000 Doesn't begin on stripe unit
    0000100 Doesn't end   on stripe unit
    0000010 Doesn't begin on stripe width
    0000001 Doesn't end   on stripe width

Yup, all blocks are 2MB aligned.

IOWs, what you are seeing is trying to do a very large allocation on a very small (8GB) XFS filesystem. It's rare someone asks to allocate >25% of the filesystem space in one allocation, so it's not surprising it triggers ENOSPC-like algorithms because it doesn't fit into a single AG....

We can probably look to optimise this, but I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Remember: stripe unit allocation alignment is a hint in XFS that we can and do ignore when necessary - it's not a binding rule.

Cheers,

Dave.

Signature · 6 lines
Dave Chinner
[email protected]
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#2 lore raw

On Sat, 2018-01-13 at 08:19 +1100, Dave Chinner wrote:
:

Quoted 6 lines

IOWs, what you are seeing is trying to do a very large allocation on a very small (8GB) XFS filesystem. It's rare someone asks to allocate >25% of the filesystem space in one allocation, so it's not surprising it triggers ENOSPC-like algorithms because it doesn't fit into a single AG....

Quoted 7 lines

We can probably look to optimise this, but I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Remember: stripe unit allocation alignment is a hint in XFS that we can and do ignore when necessary - it's not a binding rule.

Thanks for the clarification! Can XFS allocate smaller extents so that each extent will fit to an AG? ext4 creates multiple smaller extents for the same request.

-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#3 lore raw
Quoted 13 lines · Toshi

On Sat, 2018-01-13 at 08:19 +1100, Dave Chinner wrote:
:

Quoted 11 lines

IOWs, what you are seeing is trying to do a very large allocation on a very small (8GB) XFS filesystem. It's rare someone asks to allocate >25% of the filesystem space in one allocation, so it's not surprising it triggers ENOSPC-like algorithms because it doesn't fit into a single AG....

We can probably look to optimise this, but I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Quoted 5 lines
Quoted 2 lines

Remember: stripe unit allocation alignment is a hint in XFS that we can and do ignore when necessary - it's not a binding rule.

Thanks for the clarification! Can XFS allocate smaller extents so that each extent will fit to an AG?

I've already answered that question:

I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Quoted 1 line

ext4 creates multiple smaller extents for the same request.

Yes, because it has much, much smaller block groups so "allocation > max extent size (128MB)" is a common path.

It's not a common path on XFS - filesystems (and hence AGs) are typically orders of magnitude larger than the maximum extent size (8GB) so the problem only shows up when we're near ENOSPC. XFS is really not optimised for tiny filesystems, and when it comes to pmem we were lead to beleive we'd have mutliple terabytes of pmem in systems by now, not still be stuck with 8GB NVDIMMS. Hence we've spent very little time worrying about such issues because we weren't aiming to support such small capcities for very long...

Cheers,

Dave.

Signature · 6 lines
Dave Chinner
[email protected]
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#4 lore raw
Quoted 20 lines · Dave Chinner
Quoted 13 lines · Toshi

On Sat, 2018-01-13 at 08:19 +1100, Dave Chinner wrote:
:

Quoted 11 lines

IOWs, what you are seeing is trying to do a very large allocation on a very small (8GB) XFS filesystem. It's rare someone asks to allocate >25% of the filesystem space in one allocation, so it's not surprising it triggers ENOSPC-like algorithms because it doesn't fit into a single AG....

We can probably look to optimise this, but I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Quoted 5 lines
Quoted 2 lines

Remember: stripe unit allocation alignment is a hint in XFS that we can and do ignore when necessary - it's not a binding rule.

Thanks for the clarification! Can XFS allocate smaller extents so that each extent will fit to an AG?

Quoted 6 lines

I've already answered that question:

I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Right. I was thinking to limit the extent size (i.e. a half or quarter of AG size) regardless of the ENOSPC condition, but it may be the same thing.

Quoted 5 lines
Quoted 1 line

ext4 creates multiple smaller extents for the same request.

Yes, because it has much, much smaller block groups so "allocation > max extent size (128MB)" is a common path.

Quoted 8 lines

It's not a common path on XFS - filesystems (and hence AGs) are typically orders of magnitude larger than the maximum extent size (8GB) so the problem only shows up when we're near ENOSPC. XFS is really not optimised for tiny filesystems, and when it comes to pmem we were lead to beleive we'd have mutliple terabytes of pmem in systems by now, not still be stuck with 8GB NVDIMMS. Hence we've spent very little time worrying about such issues because we weren't aiming to support such small capcities for very long...

I see. Yes, there will be multiple terabytes capacity, but it will also allow to divide it into multiple smaller namespaces. So, user may continue to have relatively smaller namespaces for their use cases. If user allocates a namespace that is just big enough to host several active files, it may hit this issue regardless of their size.

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#5 lore raw
Quoted 46 lines · Toshi
Quoted 20 lines · Dave Chinner
Quoted 13 lines · Toshi

On Sat, 2018-01-13 at 08:19 +1100, Dave Chinner wrote:
:

Quoted 11 lines

IOWs, what you are seeing is trying to do a very large allocation on a very small (8GB) XFS filesystem. It's rare someone asks to allocate >25% of the filesystem space in one allocation, so it's not surprising it triggers ENOSPC-like algorithms because it doesn't fit into a single AG....

We can probably look to optimise this, but I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Quoted 5 lines
Quoted 2 lines

Remember: stripe unit allocation alignment is a hint in XFS that we can and do ignore when necessary - it's not a binding rule.

Thanks for the clarification! Can XFS allocate smaller extents so that each extent will fit to an AG?

Quoted 6 lines

I've already answered that question:

I'm not sure if we can easily differentiate this case (i.e. allocation request larger than continguous free space) from the same situation near ENOSPC when we really do have to trim to fit...

Right. I was thinking to limit the extent size (i.e. a half or quarter of AG size) regardless of the ENOSPC condition, but it may be the same thing.

Quoted 5 lines
Quoted 1 line

ext4 creates multiple smaller extents for the same request.

Yes, because it has much, much smaller block groups so "allocation > max extent size (128MB)" is a common path.

Quoted 8 lines

It's not a common path on XFS - filesystems (and hence AGs) are typically orders of magnitude larger than the maximum extent size (8GB) so the problem only shows up when we're near ENOSPC. XFS is really not optimised for tiny filesystems, and when it comes to pmem we were lead to beleive we'd have mutliple terabytes of pmem in systems by now, not still be stuck with 8GB NVDIMMS. Hence we've spent very little time worrying about such issues because we weren't aiming to support such small capcities for very long...

Quoted 5 lines

I see. Yes, there will be multiple terabytes capacity, but it will also allow to divide it into multiple smaller namespaces. So, user may continue to have relatively smaller namespaces for their use cases. If user allocates a namespace that is just big enough to host several active files, it may hit this issue regardless of their size.

I am curious, why not just give XFS all the space and let it manage the space?

--D

Quoted 2 lines

Thanks,
-Toshi

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#6 lore raw
Quoted 1 line · Darrick J. Wong

On Fri, Jan 12, 2018 at 11:15:00PM +0000, Kani, Toshi wrote:

:

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 1 line

ext4 creates multiple smaller extents for the same request.

Yes, because it has much, much smaller block groups so "allocation > max extent size (128MB)" is a common path.

It's not a common path on XFS - filesystems (and hence AGs) are typically orders of magnitude larger than the maximum extent size (8GB) so the problem only shows up when we're near ENOSPC. XFS is really not optimised for tiny filesystems, and when it comes to pmem we were lead to beleive we'd have mutliple terabytes of pmem in systems by now, not still be stuck with 8GB NVDIMMS. Hence we've spent very little time worrying about such issues because we weren't aiming to support such small capcities for very long...

Quoted 7 lines
Quoted 5 lines

I see. Yes, there will be multiple terabytes capacity, but it will also allow to divide it into multiple smaller namespaces. So, user may continue to have relatively smaller namespaces for their use cases. If user allocates a namespace that is just big enough to host several active files, it may hit this issue regardless of their size.

I am curious, why not just give XFS all the space and let it manage the space?

Well, I am not sure if having multiple namespaces would be popular use cases. But it could be useful when a system hosts multiple guests or containers that require isolation in storage space.

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm