Skip to contentSkip to conversation

Fiemap, an extent mapping ioctl

RFC PATCH In review Mark Fasheh 10 participants 35 messages May 25, 2008 – May 30, 2008 VFS

lore
Share & tools
lore
#0 lore raw

cover Fiemap, an extent mapping ioctl

Hello,

The following patches are the latest attempt at implementing a
fiemap ioctl, which can be used by userspace software to get extent information for an inode in an efficient manner.

These patches are against Linus' latest tree. While the core vfs
patch seems to be approaching feature-completeness, most of the series should still be considered as being incomplete. The fs patches in particular need some more attention. I think there's enough here however, that it makes sense to start posting to fsdevel for general comments.

Testing so far has been light, typically consisting of me running a
bare-bones ioctl wrapper program by hand:

http://www.kernel.org/pub/linux/kernel/people/mfasheh/fiemap/tests/

We definitely need some more rigorous testing software, which I
believe Eric is working on. Additionally, a port of the 'filefrag' application still needs to be completed.

A lot has changed since the last fiemap patch was posted. Mostly,
the vfs<->fs api is more fleshed out, with suitable abstractions and helper functions to aid implementation of ->fiemap. Some checks were added in the vfs patch to catch things like overflow, fs limits checks, etc. Automatic trimming of the request happens now so the fs doesn't have to worry about ranges being larger than it can handle.

Some changes were also made to the user API with the goal of
simplifying things so that it was easier for client file systems to implement a callback. My hope is that a simpler API means file systems will provide ->fiemap() quicker, and will be less likely to return results that are wrong, or worse, slightly different from other implementations.

- Except for 'fm_flags', the various in/out fields on struct fiemap got turned into a single 'out' field - the number of mapped extents (fm_mapped_extents). This gives the kernel side dealing with struct fiemap fewer 'moving parts' to deal with.

- Extent flags were cleaned up, and some new ones got added.

- Instead of forcing the user to add up all extent lengths before a given one to figure it's logical offset, an 'fe_logical" field was added to fiemap_extent. This is a lot more obvious and straight forward in my opinion, and is well worth the tradeoff of a few bytes. It also obviates the need to describe holes as their existence is easily implied now. Also, fm_start and fm_length no longer have to be 'out' variables, which goes back to the 1st listed change.

- Handling of incompatible flags was simplified to just return -EBADR and the set of not-understood flags in fm_flags.

- Documentation/filesystems/fiemap.txt has been added in the 1st patch.

Below this I will include the contents of fiemap.txt to make it more convenient for folks to get details on the API.

	--Mark

Fiemap Ioctl
============

The fiemap ioctl is an efficient method for userspace to get file extent mappings. Instead of block-by-block mapping (such as bmap), fiemap returns a list of extents.

Request Basics
--------------

A fiemap request is encoded within struct fiemap:

struct fiemap {
__u64 fm_start; /* logical offset (inclusive) at
* which to start mapping (in) */
__u64 fm_length; /* logical length of mapping which
* userspace cares about (in) */
__u32 fm_flags; /* FIEMAP_FLAG_* flags for request (in) */
__u32 fm_extent_count; /* size of fm_extents array (in) */
__u32 fm_mapped_extents; /* number of extents that were
* mapped (out) */
__u32 fm_reserved;
struct fiemap_extent fm_extents[0];
};

fm_start, and fm_length specify the logical range within the file which the process would like mappings for. Extents returned mirror those on disk - that is, the logical offset of the 1st returned extent may start before fm_start, and the range covered by the last returned extent may end after fm_length. All offsets and lengths are in bytes.

Certain flags to modify the way in which mappings are looked up can be set in fm_flags. If the kernel doesn't understand some particular flags, it will return EBADR and the contents of fm_flags will contain the set of flags which caused the error. If the kernel is compatible with all flags passed, the contents of fm_flags will be unmodified. It is up to userspace to determine whether rejection of a particular flag is fatal to it's operation. This scheme is intended to allow the fiemap interface to grow in the future but without losing compatibility with old software.

Currently, there are four flags which can be set in fm_flags:

* FIEMAP_FLAG_NUM_EXTENTS
If this flag is set, extent information will not be returned via the fm_extents array and the value of fm_extent_count will be ignored. Instead, the total number of extents covering the range will be returned via fm_mapped_extents. This is useful for programs which only want to count the number of extents in a file, but don't care about the actual extent layout.

* FIEMAP_FLAG_SYNC
If this flag is set, the kernel will sync the file before mapping extents.

* FIEMAP_FLAG_HSM_READ
If the extent is offline, retrieve it before mapping and do not flag it as FIEMAP_EXTENT_SECONDARY. This flag has no effect if the file system does not support HSM.

* FIEMAP_FLAG_XATTR
If this flag is set, the extents returned will describe the inodes extended attribute lookup tree, instead of it's data tree.

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

Extent Mapping
--------------

Note that all of this is ignored if FIEMAP_FLAG_NUM_EXTENTS is set.

Extent information is returned within the embedded fm_extents array which userspace must allocate along with the fiemap structure. The total number of fiemap_extents available should be passed via fm_extent_count. The of extents mapped by kernel will be returned via fm_mapped_extents. If the number of fiemap_extents allocated is less than would be required to map the requested range, the maximum number of extents that can be mapped in available memory will be returned and fm_mapped_extents will be equal to fm_extent_count. In that case, the last extent in the array will not complete the requested range and will not have the FIEMAP_EXTENT_LAST flag set (see the next section on extent flags).

Each extent is described by a single fiemap_extent structure as returned in fm_extents.

struct fiemap_extent {
__u64 fe_logical;/* logical offset in bytes for the start of
* the extent */
__u64 fe_physical; /* physical offset in bytes for the start
* of the extent */
__u64 fe_length; /* length in bytes for the extent */
__u32 fe_flags; /* returned FIEMAP_EXTENT_* flags for the extent */
__u32 fe_lun; /* logical device number for extent (starting at 0)*/
};

All offsets and lengths are in bytes and mirror those on disk - it is valid for an extents logical offset to start before the request or it's logical length to extend past the request. Unless FIEMAP_EXTENT_NOT_ALIGNED is returned, fe_logical, fe_physical and fe_length will be aligned to the block size of the file system.

The fe_flags field contains flags which describe the extent returned. A special flag, FIEMAP_EXTENT_LAST is always set on the last extent in the file so that the process making fiemap calls can determine when no more extents are available.

Some flags are intentionally vague and will always be set in the presence of other more specific flags. This way a program looking for a general property does not have to know all existing and future flags which imply that property.

For example, if FIEMAP_EXTENT_DATA_INLINE or FIEMAP_EXTENT_DATA_TAIL are set, FIEMAP_EXTENT_NOT_ALIGNED will also be set. A program looking for inline or tail-packed data can key on the specific flag. Software which simply cares not to try operating on non-aligned extents however, can just key on FIEMAP_EXTENT_NOT_ALIGNED, and not have to worry about all present and future flags which might imply unaligned data. Note that the opposite is not true - it would be valid for FIEMAP_EXTENT_NOT_ALIGNED to appear alone.

* FIEMAP_EXTENT_LAST
This is the last extent in the file. A mapping attempt past this extent will return nothing.

* FIEMAP_EXTENT_UNKNOWN
The location of this extent is currently unknown. This may indicate the data is stored on an inaccessible volume or that no storage has been allocated for the file yet.

* FIEMAP_EXTENT_SECONDARY
- This will also set FIEMAP_EXTENT_UNKNOWN.
The data for this extent is in secondary storage.

* FIEMAP_EXTENT_DELALLOC
- This will also set FIEMAP_EXTENT_UNKNOWN.
Delayed allocation - while there is data for this extent, it's physical location has not been allocated yet.

* FIEMAP_EXTENT_NO_DIRECT
Direct access to the data in this extent is illegal or will have undefined results.

* FIEMAP_EXTENT_NET
- This will also set FIEMAP_EXTENT_NO_DIRECT
The data for this extent is not stored in a locally-accessible device.

* FIEMAP_EXTENT_DATA_COMPRESSED
- This will also set FIEMAP_EXTENT_NO_DIRECT
The data in this extent has been compressed by the file system.

* FIEMAP_EXTENT_DATA_ENCRYPTED
- This will also set FIEMAP_EXTENT_NO_DIRECT
The data in this extent has been encrypted by the file system.

* FIEMAP_EXTENT_NOT_ALIGNED
Extent offsets and length are not guaranteed to be block aligned.

* FIEMAP_EXTENT_DATA_INLINE
This will also set FIEMAP_EXTENT_NOT_ALIGNED
Data is located within a meta data block.

* FIEMAP_EXTENT_DATA_TAIL
This will also set FIEMAP_EXTENT_NOT_ALIGNED
Data is packed into a block with data from other files.

* FIEMAP_EXTENT_UNWRITTEN
Unwritten extent - the extent is allocated but it's data has not been initialized.

VFS -> File System Implementation
---------------------------------

File systems wishing to support fiemap must implement a ->fiemap callback (on struct inode_operations):

struct inode_operations {
       ...

       int (*fiemap) (struct inode *, struct fiemap_extent_info *, u64 start,
       	   	      u64 len);

->fiemap is passed struct fiemap_extent_info which describes the fiemap request:

struct fiemap_extent_info {
unsigned int fi_flags; /* Flags as passed from user */
unsigned int fi_extents_mapped; /* Number of mapped extents */
unsigned int fi_extents_max; /* Size of fiemap_extent array */
char *fi_extents_start; /* Start of fiemap_extent array */
};

It is intended that the file system should only need to access fi_flags directly. Aside from checking fi_flags to modify callback behavior, flags which the file system can not handle, can be written into fieinfo->fi_flags. In this case, the file system *must* return -EBADR so that ioctl_fiemap() can write them into the userspace buffer.

For each extent in the request range, the file system should call the helper function, fiemap_fill_next_extent():

int fiemap_fill_next_extent(struct fiemap_extent_info *info, u64 logical,

			    u64 phys, u64 len, u32 flags, u32 lun);

fiemap_fill_next_extent() will use the passed values to populate the next free extent in the fm_extents array. 'General' extent flags will automatically be set from specific flags on behalf of the calling file system so that the userspace API is not broken.

fiemap_fill_next_extent() returns 0 on success, and 1 when the user-supplied fm_extents array is full. If an error is encountered while copying the extent to user memory, -EFAULT will be returned.

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

#1 lore raw
Quoted 4 lines · Mark Fasheh

* FIEMAP_FLAG_HSM_READ
If the extent is offline, retrieve it before mapping and do not flag it as FIEMAP_EXTENT_SECONDARY. This flag has no effect if the file system does not support HSM.

Given that there's no HSM support in mainline this should not be added. It'll be useful once we add proper HSM support, though :)

Quoted 5 lines

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

A LUN doesn't make any sense in filesystem context. That's a scsi-centric acronym that doesn't even make sense in a scsi-centric filesystem universe because a LUN can of course contain multiple partitions. It's also extremly ill-defined when using volume managers.

There's also no filesystems that actually support a single file on multiple device in mainline, the only filesystem that supports multiple data devices at all (XFS) requires each file to be on a single device.

Once we have a filesystem with real multiple data device support like btrfs or a future XFS version we can worry about this and defined a different ioctl for it.

Quoted 4 lines

Each extent is described by a single fiemap_extent structure as returned in fm_extents.

Quoted 8 lines

struct fiemap_extent {
__u64 fe_logical;/* logical offset in bytes for the start of
* the extent */
__u64 fe_physical; /* physical offset in bytes for the start
* of the extent */
__u64 fe_length; /* length in bytes for the extent */
__u32 fe_flags; /* returned FIEMAP_EXTENT_* flags for the extent */
__u32 fe_lun; /* logical device number for extent (starting at 0)*/

Again this lun thing is horribly ill-defined. There is no such thing as a logic device number in our filesystem terminology.

Quoted 6 lines

struct fiemap_extent_info {
unsigned int fi_flags; /* Flags as passed from user */
unsigned int fi_extents_mapped; /* Number of mapped extents */
unsigned int fi_extents_max; /* Size of fiemap_extent array */
char *fi_extents_start; /* Start of fiemap_extent array */
};

Why is this passes a structure instead of individual arguments? Also why isn't fi_extents_start properly typed?

Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

Just define a simple FIECOUNT ioctl.

#2 lore raw
Quoted 32 lines · Christoph Hellwig
Quoted 5 lines

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

A LUN doesn't make any sense in filesystem context. That's a scsi-centric acronym that doesn't even make sense in a scsi-centric filesystem universe because a LUN can of course contain multiple partitions. It's also extremly ill-defined when using volume managers.

There's also no filesystems that actually support a single file on multiple device in mainline, the only filesystem that supports multiple data devices at all (XFS) requires each file to be on a single device.

Once we have a filesystem with real multiple data device support like btrfs or a future XFS version we can worry about this and defined a different ioctl for it.

Quoted 4 lines

Each extent is described by a single fiemap_extent structure as returned in fm_extents.

Quoted 8 lines

struct fiemap_extent {
__u64 fe_logical;/* logical offset in bytes for the start of
* the extent */
__u64 fe_physical; /* physical offset in bytes for the start
* of the extent */
__u64 fe_length; /* length in bytes for the extent */
__u32 fe_flags; /* returned FIEMAP_EXTENT_* flags for the extent */
__u32 fe_lun; /* logical device number for extent (starting at 0)*/

Quoted 2 lines

Again this lun thing is horribly ill-defined. There is no such thing as a logic device number in our filesystem terminology.

The vxfs filesystem is capable of having each extent in a file on a different device. However, I don't think freevxfs supports that.

I do agree that calling each one a "lun" is not really meaningful.

Brad Boyer
[email protected]

#3 lore raw
In reply to Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Mark Fasheh

* FIEMAP_FLAG_HSM_READ
If the extent is offline, retrieve it before mapping and do not flag it as FIEMAP_EXTENT_SECONDARY. This flag has no effect if the file system does not support HSM.

Given that there's no HSM support in mainline this should not be added. It'll be useful once we add proper HSM support, though :)

This was added at the request of David for XFS, because the XFS bmap ioctl defaults to reading in extents from HSM. I don't have any attachment to it myself.

Quoted 10 lines
Quoted 5 lines

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

A LUN doesn't make any sense in filesystem context. That's a scsi-centric acronym that doesn't even make sense in a scsi-centric filesystem universe because a LUN can of course contain multiple partitions. It's also extremly ill-defined when using volume managers.

What else do you propose calling this? It isn't a LUN in the SCSI sense of course, but there is definitely a need to be able to identify multiple disks. Regardless of whether there is a single disk or multiple disks involved, it is generally called a LUN. It is a better than calling it a "disk" or a "partition".

Quoted 7 lines

There's also no filesystems that actually support a single file on multiple device in mainline, the only filesystem that supports multiple data devices at all (XFS) requires each file to be on a single device.

Once we have a filesystem with real multiple data device support like btrfs or a future XFS version we can worry about this and defined a different ioctl for it.

I don't see why we need a different ioctl for mapping extents on a filesystem that support direct access to multiple disks. Having one mechanism that returns the file mapping is much more simple for user space applications (filefrag, cp, tar, gzip, etc) than having to use different ioctls for different backing filesystems.

Quoted 12 lines
Quoted 11 lines

Each extent is described by a single fiemap_extent structure as returned in fm_extents.

struct fiemap_extent {
__u64 fe_logical;/* logical offset in bytes for the start of
* the extent */
__u64 fe_physical; /* physical offset in bytes for the start
* of the extent */
__u64 fe_length; /* length in bytes for the extent */
__u32 fe_flags; /* returned FIEMAP_EXTENT_* flags for the extent */
__u32 fe_lun; /* logical device number for extent (starting at 0)*/

Quoted 2 lines

Again this lun thing is horribly ill-defined. There is no such thing as a logic device number in our filesystem terminology.

Propose a better name then, but the need for it will not go away. This is needed for Lustre, btrfs, pNFS, etc. The whole point of developing this API and getting input from all of the main filesystems was to have a single common interface that could be used by all filesystems.

Quoted 8 lines
Quoted 6 lines

struct fiemap_extent_info {
unsigned int fi_flags; /* Flags as passed from user */
unsigned int fi_extents_mapped; /* Number of mapped extents */
unsigned int fi_extents_max; /* Size of fiemap_extent array */
char *fi_extents_start; /* Start of fiemap_extent array */
};

Why is this passes a structure instead of individual arguments?

Saves on passing this around as arguments on the stack? Also, for ext4 there is an iterator function which needs a private data struct passed, and it doesn't make sense to require duplicating all of this information again.

Quoted 1 line

Also why isn't fi_extents_start properly typed?

I was wondering about that, I'm not sure why Mark implemented it that way. I would have thought that it should be a struct fiemap_extent *. I thought maybe to allow for misaligned userspace pointers, but I'm not sure.

Quoted 6 lines
Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

I don't see that at all. The operations that the filesystem has to do are basically the same whether it is counting extents or returning them. All that would result from having separate ioctl and filesystem methods would be a lot of code duplication.

The fiemap_fill_next_extents() call will handle the NUM_EXTENTS operation internally, and the filesystem code doesn't need to special case this at all. The only time the NUM_EXTENTS case would be handled by the filesystem specially would be if it tracks the count of extents itself for some reason.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#4 lore raw
Quoted 11 lines · Andreas Dilger
Quoted 4 lines · Christoph Hellwig

A LUN doesn't make any sense in filesystem context. That's a scsi-centric acronym that doesn't even make sense in a scsi-centric filesystem universe because a LUN can of course contain multiple partitions. It's also extremly ill-defined when using volume managers.

What else do you propose calling this? It isn't a LUN in the SCSI sense of course, but there is definitely a need to be able to identify multiple disks. Regardless of whether there is a single disk or multiple disks involved, it is generally called a LUN. It is a better than calling it a "disk" or a "partition".

How about "device"? It's more generic than LUN but also doesn't imply a particular implementation. A volume manager exports individual volumes as devices. Regular partitions on a disk are also exported to user space as devices. I don't think anyone would get confused and think a filesystem would be using a non-storage device.

Brad Boyer
[email protected]

#5 lore raw
In reply to Andreas Dilger
Quoted 10 lines · Andreas Dilger
Quoted 4 lines

A LUN doesn't make any sense in filesystem context. That's a scsi-centric acronym that doesn't even make sense in a scsi-centric filesystem universe because a LUN can of course contain multiple partitions. It's also extremly ill-defined when using volume managers.

What else do you propose calling this? It isn't a LUN in the SCSI sense of course, but there is definitely a need to be able to identify multiple disks. Regardless of whether there is a single disk or multiple disks involved, it is generally called a LUN. It is a better than calling it a "disk" or a "partition".

See below because the naming really depends on defining the semantics of the this field.

Quoted 5 lines

I don't see why we need a different ioctl for mapping extents on a filesystem that support direct access to multiple disks. Having one mechanism that returns the file mapping is much more simple for user space applications (filefrag, cp, tar, gzip, etc) than having to use different ioctls for different backing filesystems.

Well, we could add a dev field that contains the dev_t for the underlying block device. That would work for the current XFS realtime device aswell as for my work to map different XFS AGs to different devices. It wouldn't work for btrfs with integrated raid code where a single extent can span multiple underlying devices, the same probably applies to pnfs.

Quoted 6 lines
Quoted 1 line

Why is this passes a structure instead of individual arguments?

Saves on passing this around as arguments on the stack? Also, for ext4 there is an iterator function which needs a private data struct passed, and it doesn't make sense to require duplicating all of this information again.

Ok.

Quoted 12 lines
Quoted 6 lines
Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

I don't see that at all. The operations that the filesystem has to do are basically the same whether it is counting extents or returning them. All that would result from having separate ioctl and filesystem methods would be a lot of code duplication.

Quoted 5 lines

The fiemap_fill_next_extents() call will handle the NUM_EXTENTS operation internally, and the filesystem code doesn't need to special case this at all. The only time the NUM_EXTENTS case would be handled by the filesystem specially would be if it tracks the count of extents itself for some reason.

It just special cases it. As does for example the ext4 handler. Please keep the API simple instead of overloading it already from the start. And when you look at the ext4 implementation with the extent walker it's pretty simple to implement a fiecount by having a second callback with a trivial shared helper.

#6 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines

I don't see why we need a different ioctl for mapping extents on a filesystem that support direct access to multiple disks. Having one mechanism that returns the file mapping is much more simple for user space applications (filefrag, cp, tar, gzip, etc) than having to use different ioctls for different backing filesystems.

Quoted 6 lines

Well, we could add a dev field that contains the dev_t for the underlying block device. That would work for the current XFS realtime device aswell as for my work to map different XFS AGs to different devices. It wouldn't work for btrfs with integrated raid code where a single extent can span multiple underlying devices, the same probably applies to pnfs.

Dev_t seems reasonable to me.

Anything any more complicated than just wanting a simple device identifier needs a more dedicated API.

	--Mark

--
Mark Fasheh

#7 lore raw
In reply to Christoph Hellwig

On Sunday 25 May 2008, Christoph Hellwig wrote:

Thanks for doing this Mark ;)

Quoted 9 lines · Mark Fasheh
Quoted 4 lines

* FIEMAP_FLAG_HSM_READ
If the extent is offline, retrieve it before mapping and do not flag it as FIEMAP_EXTENT_SECONDARY. This flag has no effect if the file system does not support HSM.

Given that there's no HSM support in mainline this should not be added. It'll be useful once we add proper HSM support, though :)

The HSM flag doesn't hurt, and it allows the people actually shipping hsm patches to use fiemap without extending the api themselves. Reserving the flag isn't a bad idea.

Quoted 11 lines
Quoted 5 lines

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

A LUN doesn't make any sense in filesystem context. That's a scsi-centric acronym that doesn't even make sense in a scsi-centric filesystem universe because a LUN can of course contain multiple partitions. It's also extremly ill-defined when using volume managers.

Quoted 8 lines

There's also no filesystems that actually support a single file on multiple device in mainline, the only filesystem that supports multiple data devices at all (XFS) requires each file to be on a single device.

Once we have a filesystem with real multiple data device support like btrfs or a future XFS version we can worry about this and defined a different ioctl for it.

For btrfs I would return the logical extents via fiemap (just like the file were on lvm) and make btrfs specific ioctls for details about where the file actually lived.

fiemap alone isn't a great way to describe raid levels or complex storage topologies. To include physical information I would also have to encode the raid level used and information about all the devices the data is replicated on (raid1/10)

-chris

#8 lore raw
Quoted 9 lines · Chris Mason

On Sunday 25 May 2008, Christoph Hellwig wrote:

Thanks for doing this Mark ;)

Quoted 5 lines · Mark Fasheh
Quoted 4 lines

* FIEMAP_FLAG_HSM_READ
If the extent is offline, retrieve it before mapping and do not flag it as FIEMAP_EXTENT_SECONDARY. This flag has no effect if the file system does not support HSM.

Quoted 7 lines
Quoted 3 lines

Given that there's no HSM support in mainline this should not be added. It'll be useful once we add proper HSM support, though :)

The HSM flag doesn't hurt, and it allows the people actually shipping hsm patches to use fiemap without extending the api themselves. Reserving the flag isn't a bad idea.

Here I agree. HSM is a generic enough concept, and I think this interface's API w.r.t. HSM is well-enough defined that there's no reason not to go ahead & put it in now, IMHO.

-Eric

#9 lore raw
Quoted 7 lines · Eric Sandeen
Quoted 3 lines

The HSM flag doesn't hurt, and it allows the people actually shipping hsm patches to use fiemap without extending the api themselves. Reserving the flag isn't a bad idea.

Here I agree. HSM is a generic enough concept, and I think this interface's API w.r.t. HSM is well-enough defined that there's no reason not to go ahead & put it in now, IMHO.

But there is no such thing as HSM support anywher near mainline. Call me a dickhead, but I'm 100% against adding anything helping HSM until people get their act together to actually add HSM support. It's something really useful that we should have, and not something that should be in really grotty out of tree codebases.

#10 lore raw
Quoted 10 lines · Christoph Hellwig
Quoted 3 lines · Eric Sandeen

Here I agree. HSM is a generic enough concept, and I think this interface's API w.r.t. HSM is well-enough defined that there's no reason not to go ahead & put it in now, IMHO.

But there is no such thing as HSM support anywher near mainline. Call me a dickhead, but I'm 100% against adding anything helping HSM until people get their act together to actually add HSM support. It's something really useful that we should have, and not something that should be in really grotty out of tree codebases.

I would love to see HSM in mainline too, but this is a single
flag that belongs in a well-defined interface. It isn't the place to fight over.

Joel

Signature · 9 lines
Life's Little Instruction Book #226

	"When someone hugs you, let them be the first to let go."

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
#11 lore raw
In reply to Chris Mason

For what it is worth, a few comments from a newbie who has experience with a non-linux filesystem that has a similar API and supports files spread across multiple devices.

Quoted 6 lines · Mark Fasheh

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

This should return an error (ENOTSUPPORTED ?) if the FS does not support multiple devices OR does not support sort-by-lun-order so the caller does not count on the info being sorted. Even an FS that supports multiple devices per file may be unable to sort it by on-disk-order without consuming an ugly set of resources.

Quoted 5 lines · Christoph Hellwig
Quoted 1 line

__u32 fe_lun; /* logical device number for extent (starting at 0)*/

Again this lun thing is horribly ill-defined. There is no such thing as a logic device number in our filesystem terminology.

I agree that LUN is confusing. In my opinion the words "logical" and "number" are overused and meaningless. As Brad suggested, "device" would be preferable, or "unit", but unfortunately every word I can think of has some other definition too :)

Our term was "volume"... an awful designation.

Quoted 3 lines · Chris Mason

For btrfs I would return the logical extents via fiemap (just like the file were on lvm) and make btrfs specific ioctls for details about where the file actually lived.

fiemap alone isn't a great way to describe raid levels or complex storage topologies. To include physical information I would also have to encode the raid level used and information about all the devices the data is replicated on (raid1/10)

fiemap by itself is useful for programs that want to determine how fragmented a file is or where sparse areas are to skip.

At least one more generic API is needed to enumerate the device number to device (path name, inode, socket, ... ?). In our case this was only used for clusters.

For the complex case you describe, it might be possible to have an "enumerate" api that could be used to traverse each layer for more detail. I hope this is done generically by someone.

A final thought on this:

Quoted 1 line

__u32 fe_lun; /* logical device number for extent (starting at 0)*/

While the flags field can be used to tell the validity of this number, we found that starting at 0 was not a good practice. We started at 1 so 0 was always a not-valid. One way this can be useful is if you have delayed allocation, you can indicate "intended device" with a non-0 number. Of course other values such as max_int could be termed "invalid" instead.

Another point to document is whether this number is a contiguous series (1, 2, 3,... N) defining the location based on the current device list or is possibly a sparse (1, 2, 6) series because the FS tracks devices that have been removed. In our implementation both views were present for different consumers. The sparse series was native and the contiguous series a translation.

jim

#12 lore raw
Quoted 11 lines · jim owens

For what it is worth, a few comments from a newbie who has experience with a non-linux filesystem that has a similar API and supports files spread across multiple devices.

Quoted 5 lines · Mark Fasheh

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

Quoted 5 lines

This should return an error (ENOTSUPPORTED ?) if the FS does not support multiple devices OR does not support sort-by-lun-order so the caller does not count on the info being sorted. Even an FS that supports multiple devices per file may be unable to sort it by on-disk-order without consuming an ugly set of resources.

That's a good point, I couldn't provide 100% sorted output even if I wanted to.

Quoted 19 lines
Quoted 4 lines · Christoph Hellwig
Quoted 1 line

__u32 fe_lun; /* logical device number for extent (starting at 0)*/

Again this lun thing is horribly ill-defined. There is no such thing as a logic device number in our filesystem terminology.

I agree that LUN is confusing. In my opinion the words "logical" and "number" are overused and meaningless. As Brad suggested, "device" would be preferable, or "unit", but unfortunately every word I can think of has some other definition too :)

Our term was "volume"... an awful designation.

Quoted 4 lines · Chris Mason

For btrfs I would return the logical extents via fiemap (just like the file were on lvm) and make btrfs specific ioctls for details about where the file actually lived.

Quoted 7 lines
Quoted 4 lines

fiemap alone isn't a great way to describe raid levels or complex storage topologies. To include physical information I would also have to encode the raid level used and information about all the devices the data is replicated on (raid1/10)

fiemap by itself is useful for programs that want to determine how fragmented a file is or where sparse areas are to skip.

Yes, and since it has no concurrency semantics, use outside of that quickly gets difficult. fibmap is used by lilo, and reiserfs needs a special ioctl that said i've-called-fibmap-please-don't-move-these-bytes that prevented tail packing.

Quoted 9 lines

At least one more generic API is needed to enumerate the device number to device (path name, inode, socket, ... ?). In our case this was only used for clusters.

For the complex case you describe, it might be possible to have an "enumerate" api that could be used to traverse each layer for more detail. I hope this is done generically by someone.

It would be especially interesting if the enumerate API actually went all the way down to the lvm/md layers as well.

Quoted 9 lines

A final thought on this:

Quoted 1 line

__u32 fe_lun; /* logical device number for extent (starting at 0)*/

While the flags field can be used to tell the validity of this number, we found that starting at 0 was not a good practice. We started at 1 so 0 was always a not-valid. One way this can be useful is if you have delayed allocation, you can indicate "intended device" with a non-0 number. Of course other values such as max_int could be termed "invalid" instead.

I use 0 as not-valid as well. The original intent was 0 meant logical-block-number, signaling additional lookups were needed. But I haven't found a good use case for that yet.

Quoted 7 lines

Another point to document is whether this number is a contiguous series (1, 2, 3,... N) defining the location based on the current device list or is possibly a sparse (1, 2, 6) series because the FS tracks devices that have been removed. In our implementation both views were present for different consumers. The sparse series was native and the contiguous series a translation.

Interesting, I've been presenting the sparse representation only.

-chris

#13 lore raw
Quoted 12 lines · Chris Mason
Quoted 11 lines · jim owens

For what it is worth, a few comments from a newbie who has experience with a non-linux filesystem that has a similar API and supports files spread across multiple devices.

Quoted 5 lines · Mark Fasheh

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

Quoted 8 lines
Quoted 5 lines

This should return an error (ENOTSUPPORTED ?) if the FS does not support multiple devices OR does not support sort-by-lun-order so the caller does not count on the info being sorted. Even an FS that supports multiple devices per file may be unable to sort it by on-disk-order without consuming an ugly set of resources.

That's a good point, I couldn't provide 100% sorted output even if I wanted to.

I'm OK with this also. The only reason I thought "simple" filesystems (i.e. single-lun) should ignore FLAG_LUN_ORDER is so that tools like filefrag can always try with LUN_ORDER and in most cases still get a mapping returned. If the filesystem doesn't care about LUN_MAPPING, no harm done, because all of the extents live on a single LUN anyways. If a multi-device filesystem doesn't want to implement LUN_ORDER, returning -EBADR is perfectly acceptable because the application will retry without the unsupported flags (LUN_ORDER in this case) and get the logical file offset order data returned.

For Lustre, it is completely inefficient to return data in non-LUN_ORDER, because it is doing RAID-0 striping of the file data across data servers. A 100MB 2-stripe file with 1MB stripes would have to return 100 extents, even if the file data is allocated contiguously on disk in the backing filesystems in two 50MB chunks. With LUN_ORDER it will return 2 extents and the user can see much more clearly that the file is layed out well.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines · Christoph Hellwig
Quoted 1 line

__u32 fe_lun; /* logical device number for extent (starting at 0)*/

Again this lun thing is horribly ill-defined. There is no such thing as a logic device number in our filesystem terminology.

I agree that LUN is confusing. In my opinion the words "logical" and "number" are overused and meaningless. As Brad suggested, "device" would be preferable, or "unit", but unfortunately every word I can think of has some other definition too :)

Calling it "device" instead of "LUN" for this is fine...

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines · Christoph Hellwig

Well, we could add a dev field that contains the dev_t for the underlying block device. That would work for the current XFS realtime device aswell as for my work to map different XFS AGs to different devices. It wouldn't work for btrfs with integrated raid code where a single extent can span multiple underlying devices, the same probably applies to pnfs.

... but I don't think it should necessarily be _required_ to return a real "dev_t" (major, minor) device. For network filesystems this is meaningless. If it is possible for FIEMAP_EXTENT_NET to signal that the device is not a local/physical device (where a dev_t has no meaning), and simply allow an enumeration [0, 1, 2, ...] of the logical devices then I think this is reasonable. The mapping of logical devices to servers is available separately with a Lustre-specific ioctl.

This passes more information for filesystems that have local devices while not breaking the functionality for network filesystems and could be used as an efficient replacement for lilo's use of FIBMAP.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines · Chris Mason

For btrfs I would return the logical extents via fiemap (just like the file were on lvm) and make btrfs specific ioctls for details about where the file actually lived.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

fiemap alone isn't a great way to describe raid levels or complex storage topologies. To include physical information I would also have to encode the raid level used and information about all the devices the data is replicated on (raid1/10)

fiemap by itself is useful for programs that want to determine how fragmented a file is or where sparse areas are to skip.

For RAID1/10 you can return multiple logical->physical extent mappings for the same logical range of the file with different "device" IDs. You could do the same for RAID5 returning each of the data and parity chunks with "NO_DIRECT" if desired (maybe only on the parity extent, or don't return the parity extent at all). The spec does not require that the returned extents be non-overlapping.

In fact Mark, Eric, and I were discussing the ability to request mappings for metadata blocks in addition to the data blocks. The metadata blocks would also overlap the data blocks (with FLAG_METADATA set in the metadata extent) so that it is possible to return to the client (if requested) the inode block with [0-EOF] mapping, indirect blocks with their corresponding data mappings, and the file data blocks.

This came up in the context of ext4 trying to visualize different metadata placement algorithms and would be very useful information. It might also be useful for filesystem defragmentation utilities.

Quoted 4 lines

Yes, and since it has no concurrency semantics, use outside of that quickly gets difficult. fibmap is used by lilo, and reiserfs needs a special ioctl that said i've-called-fibmap-please-don't-move-these-bytes that prevented tail packing.

Wasn't that turned into an ext3-like SETFLAGS ioctl for "NOTAIL" on the inode?

My point of view is that FIEMAP is a file layout visualization API that could also be used in certain cases for direct data access. Since any direct access of data returned by FIEMAP is inherently racy (as is FIBMAP), I'm less concerned with the mappings being fully consistent, and more concerned with providing the maximum amount of information.

Any application using FIEMAP for direct data access (e.g. dump of some kind) either has to guard against races itself by verifying the mapping again afterward, or for uses like lilo trust that the admin is doing the right thing. That isn't a new issue with FIEMAP vs FIBMAP.

Quoted 10 lines
Quoted 9 lines

A final thought on this:

Quoted 1 line

__u32 fe_lun; /* logical device number for extent (starting at 0)*/

While the flags field can be used to tell the validity of this number, we found that starting at 0 was not a good practice. We started at 1 so 0 was always a not-valid. One way this can be useful is if you have delayed allocation, you can indicate "intended device" with a non-0 number. Of course other values such as max_int could be termed "invalid" instead.

Quoted 3 lines

I use 0 as not-valid as well. The original intent was 0 meant logical-block-number, signaling additional lookups were needed. But I haven't found a good use case for that yet.

I would prefer that the fe_lun (or fe_device as is now preferred) be at least somewhat implementation-specific. For local filesystems, returning the device number seems reasonable and would mean that "0" is not a valid return value, but I'd prefer to allow this to be an index number for Lustre or other non-local filesystems, and in that case "0" be a valid device index number. Since there are already flags for unallocated and unkown extents, I don't think we should depend on fe_device == 0 to have a special meaning for network filesystems.

Quoted 8 lines
Quoted 6 lines

Another point to document is whether this number is a contiguous series (1, 2, 3,... N) defining the location based on the current device list or is possibly a sparse (1, 2, 6) series because the FS tracks devices that have been removed. In our implementation both views were present for different consumers. The sparse series was native and the contiguous series a translation.

Interesting, I've been presenting the sparse representation only.

For Lustre, the fe_lun/fe_device returned for a file extent would indicate the base-0 server index on which each of the file fragments resides. A given file will normally be striped over a subset of the data servers, so it would be normal to get extents returned that are a sparse subset of all available data servers.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#14 lore raw
Quoted 32 lines · Andreas Dilger
Quoted 12 lines · Chris Mason
Quoted 11 lines · jim owens

For what it is worth, a few comments from a newbie who has experience with a non-linux filesystem that has a similar API and supports files spread across multiple devices.

Quoted 5 lines · Mark Fasheh

* FIEMAP_FLAG_LUN_ORDER
If the file system stripes file data, this will return contiguous regions of physical allocation, sorted by LUN. Logical offsets may not make sense if this flag is passed. If the file system does not support multiple LUNs, this flag will be ignored.

Quoted 8 lines
Quoted 5 lines

This should return an error (ENOTSUPPORTED ?) if the FS does not support multiple devices OR does not support sort-by-lun-order so the caller does not count on the info being sorted. Even an FS that supports multiple devices per file may be unable to sort it by on-disk-order without consuming an ugly set of resources.

That's a good point, I couldn't provide 100% sorted output even if I wanted to.

I'm OK with this also. The only reason I thought "simple" filesystems (i.e. single-lun) should ignore FLAG_LUN_ORDER is so that tools like filefrag can always try with LUN_ORDER and in most cases still get a mapping returned. If the filesystem doesn't care about LUN_MAPPING, no harm done, because all of the extents live on a single LUN anyways. If a multi-device filesystem doesn't want to implement LUN_ORDER, returning -EBADR is perfectly acceptable because the application will retry without the unsupported flags (LUN_ORDER in this case) and get the logical file offset order data returned.

Quoted 6 lines

For Lustre, it is completely inefficient to return data in non-LUN_ORDER, because it is doing RAID-0 striping of the file data across data servers. A 100MB 2-stripe file with 1MB stripes would have to return 100 extents, even if the file data is allocated contiguously on disk in the backing filesystems in two 50MB chunks. With LUN_ORDER it will return 2 extents and the user can see much more clearly that the file is layed out well.

Ah, so lustre doesn't have a logical address layer at all? In my case the files contain pointers to contiguous logical extent and the lower layers of the FS figure out that is raid0/1/10 or whatever future crud I toss in.

If the logical extents are contiguous it is safe to assume the lower end is also contiguous.

[ huge snip ;) ]

Quoted 10 lines

My point of view is that FIEMAP is a file layout visualization API that could also be used in certain cases for direct data access. Since any direct access of data returned by FIEMAP is inherently racy (as is FIBMAP), I'm less concerned with the mappings being fully consistent, and more concerned with providing the maximum amount of information.

Any application using FIEMAP for direct data access (e.g. dump of some kind) either has to guard against races itself by verifying the mapping again afterward, or for uses like lilo trust that the admin is doing the right thing. That isn't a new issue with FIEMAP vs FIBMAP.

So, I'm a big fan of better layout visualization and creating APIs to improve it. At some point we need to take a step back and ask if those apis are better left to other tools instead of heaping them all into fiemap.

The advantage of dropping the lun support from fiemap and pushing it into a new ioctl/syscall is that we can determine the underlying storage topology for any logical block on the device, including those underneath md/dm without worrying about a backing file.

And then we can get interesting information about stripe widths, preferred IO sizes etc etc.

[ lots of other stuff that makes good sense snipped too ]

-chris

#15 lore raw
Quoted 8 lines · Chris Mason
Quoted 6 lines · Andreas Dilger

For Lustre, it is completely inefficient to return data in non-LUN_ORDER, because it is doing RAID-0 striping of the file data across data servers. A 100MB 2-stripe file with 1MB stripes would have to return 100 extents, even if the file data is allocated contiguously on disk in the backing filesystems in two 50MB chunks. With LUN_ORDER it will return 2 extents and the user can see much more clearly that the file is layed out well.

Quoted 6 lines

Ah, so lustre doesn't have a logical address layer at all? In my case the files contain pointers to contiguous logical extent and the lower layers of the FS figure out that is raid0/1/10 or whatever future crud I toss in.

If the logical extents are contiguous it is safe to assume the lower end is also contiguous.

Well, Lustre has a logical address layer on a per-file basis, but the layout maps from the file offsets to multiple object offsets. There is no "flat" logical device in the background which file allocations are coming from, because the API provided to the client is based only on objects and offsets, and there may be multiple objects that map into a single file via some striping. That is currently RAID-0 across objects, but it might be RAID-1/5/6 or something else in the future. With the RAID-0 layout, the logical file offsets round-robin across the multiple objects with a certain stripe size (default 1MB).

It sounds like you actually have the same setup with btrfs (if it is at all like ZFS) that file blocks map onto multiple disks, and there may be multiple copies of the data (RAID-1/10).

What a user/administrator really cares about in the end is whether the files are allocated contiguously within the objects on the server filesystems. If we were to run filefrag (with FIEMAP support) on a Lustre file without LUN_ORDER, or maybe a RAID-5 btrfs file, it would return a list of extents, each broken up at smaller boundaries, and it will convey the wrong idea of how the file is layed out physically.

If we run fiemap (with LUN_ORDER) what will happen is we get the larger (hopefully) extents that are actually contiguously allocated in the backing filesystems. Since this is a network object-based filesystem, we don't really care about the _actual_ file offset->device block number layout as much as the overall picture of file fragmentation and layout.

Quoted 11 lines
Quoted 10 lines

My point of view is that FIEMAP is a file layout visualization API that could also be used in certain cases for direct data access. Since any direct access of data returned by FIEMAP is inherently racy (as is FIBMAP), I'm less concerned with the mappings being fully consistent, and more concerned with providing the maximum amount of information.

Any application using FIEMAP for direct data access (e.g. dump of some kind) either has to guard against races itself by verifying the mapping again afterward, or for uses like lilo trust that the admin is doing the right thing. That isn't a new issue with FIEMAP vs FIBMAP.

Quoted 7 lines

So, I'm a big fan of better layout visualization and creating APIs to improve it. At some point we need to take a step back and ask if those apis are better left to other tools instead of heaping them all into fiemap. The advantage of dropping the lun support from fiemap and pushing it into a new ioctl/syscall is that we can determine the underlying storage topology for any logical block on the device, including those underneath md/dm without worrying about a backing file.

Argh, that would make FIEMAP basically unsuitable for a mutli-device filesystem like Lustre and pNFS and the future direction of XFS (I think), and btrfs IMHO, or even ZFS. There just isn't a single address space that files can be mapped to.

A major reason I proposed FIEMAP to linux-fsdevel in the first place, instead of just keeping it internal to Lustre and maybe ext4, is because it is generally useful interface for efficiently determining file layout information, and isn't tied to block devices like FIBMAP is.

It is useful for many different reasons like cp/tar to skip holes (not using the physical offset information, just the logical extents and flags like UNWRITTEN) to avoid reading empty parts of the file, layout visualization, maybe defrag, etc.

Dropping lun/device support, and removing all of the flexibility of the FIEMAP interface design, is IMHO killing the whole reason I proposed FIEMAP in the first place.

Quoted 2 lines

And then we can get interesting information about stripe widths, preferred IO sizes etc etc.

I agree that this part is somewhat orthogonal. In the vast majority of cases stripe width, stripe count, IO size, etc can be encapsulated into a small number of parameters and does not need to be specified on a per-block or per-extent basis. The actual layout of the file (returned by FIEMAP) is a natural consequence of these parameters, as they (may) influence the filesystem in making allocation decisions. Returning the metadata layout as part of FIEMAP makes sense to me, because it boils down to logical->{physical,device} ranges in the end. Returning the generic file layout parameters doesn't make sense, on the other hand.

Lustre (and in fact several of the HPC filesystem vendors) would like to come up with a common API (virtual xattr?) to be able to extract and restore the generic layout information but that is for a separate email.

Cheer , Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#16 lore raw
Quoted 28 lines · Andreas Dilger
Quoted 9 lines · Chris Mason
Quoted 7 lines · Andreas Dilger

For Lustre, it is completely inefficient to return data in non-LUN_ORDER, because it is doing RAID-0 striping of the file data across data servers. A 100MB 2-stripe file with 1MB stripes would have to return 100 extents, even if the file data is allocated contiguously on disk in the backing filesystems in two 50MB chunks. With LUN_ORDER it will return 2 extents and the user can see much more clearly that the file is layed out well.

Quoted 7 lines

Ah, so lustre doesn't have a logical address layer at all? In my case the files contain pointers to contiguous logical extent and the lower layers of the FS figure out that is raid0/1/10 or whatever future crud I toss in.

If the logical extents are contiguous it is safe to assume the lower end is also contiguous.

Well, Lustre has a logical address layer on a per-file basis, but the layout maps from the file offsets to multiple object offsets. There is no "flat" logical device in the background which file allocations are coming from, because the API provided to the client is based only on objects and offsets, and there may be multiple objects that map into a single file via some striping. That is currently RAID-0 across objects, but it might be RAID-1/5/6 or something else in the future. With the RAID-0 layout, the logical file offsets round-robin across the multiple objects with a certain stripe size (default 1MB).

Quoted 3 lines

It sounds like you actually have the same setup with btrfs (if it is at all like ZFS) that file blocks map onto multiple disks, and there may be multiple copies of the data (RAID-1/10).

In my case, all pointers to extents (both metadata blocks and file data) reference a logical address space. So, even for raid10 or raid5/6 if I ever code it, there is a central place that does translation from logical->physical block(s).

The disk format supports multiple (2^64) such namespaces but that isn't being used yet.

Quoted 9 lines

What a user/administrator really cares about in the end is whether the files are allocated contiguously within the objects on the server filesystems. If we were to run filefrag (with FIEMAP support) on a Lustre file without LUN_ORDER, or maybe a RAID-5 btrfs file, it would return a list of extents, each broken up at smaller boundaries, and it will convey the wrong idea of how the file is layed out physically.

For Btrfs, it'll always return the logical extents, and because the storage is grouped in relatively large chunks (~1GB or more), this is sufficiently enough for measuring fragmentation.

But, if lustre doesn't have this kind of logical backing store, I think it is reason enough to keep the lun interface. I know lots of people are against adding interfaces to the kernel for out of tree projects, but the per-file logical mapping you describe is a very reasonable way to design things, and we might as well leave it in for future use.

Quoted 3 lines

Dropping lun/device support, and removing all of the flexibility of the FIEMAP interface design, is IMHO killing the whole reason I proposed FIEMAP in the first place.

My goal isn't to remove the flexibility from the interface design, it is just to ask if all of this functionality needs to be in one ioctl. At least the device number / lun bit makes sense now (Mark, if you keep it, please don't make this a dev_t) thanks for the extra details.

-chris

#17 lore raw
In reply to Andreas Dilger
Quoted 11 lines · Andreas Dilger

... but I don't think it should necessarily be _required_ to return a real "dev_t" (major, minor) device. For network filesystems this is meaningless. If it is possible for FIEMAP_EXTENT_NET to signal that the device is not a local/physical device (where a dev_t has no meaning), and simply allow an enumeration [0, 1, 2, ...] of the logical devices then I think this is reasonable. The mapping of logical devices to servers is available separately with a Lustre-specific ioctl.

This passes more information for filesystems that have local devices while not breaking the functionality for network filesystems and could be used as an efficient replacement for lilo's use of FIBMAP.

A dev_t actually means something for the only in-tree users of this interface, so there's no point making this interface worse for some long-term out of tree code. And it's not like you simply can't allow multiple anonymous blockdevices for your networked filesystems similar to the one used for st_dev already.

Quoted 6 lines

For RAID1/10 you can return multiple logical->physical extent mappings for the same logical range of the file with different "device" IDs. You could do the same for RAID5 returning each of the data and parity chunks with "NO_DIRECT" if desired (maybe only on the parity extent, or don't return the parity extent at all). The spec does not require that the returned extents be non-overlapping.

Umm, no. That's just make the interface too complicated. I can bet with your that userspace programmers will generally only test their code with simple filesystems and hell will break lose when they get these multiple ranges. Especially as that's a very unnatural interface.

#18 lore raw
Quoted 13 lines · Christoph Hellwig
Quoted 11 lines · Andreas Dilger

... but I don't think it should necessarily be _required_ to return a real "dev_t" (major, minor) device. For network filesystems this is meaningless. If it is possible for FIEMAP_EXTENT_NET to signal that the device is not a local/physical device (where a dev_t has no meaning), and simply allow an enumeration [0, 1, 2, ...] of the logical devices then I think this is reasonable. The mapping of logical devices to servers is available separately with a Lustre-specific ioctl.

This passes more information for filesystems that have local devices while not breaking the functionality for network filesystems and could be used as an efficient replacement for lilo's use of FIBMAP.

Quoted 5 lines

A dev_t actually means something for the only in-tree users of this interface, so there's no point making this interface worse for some long-term out of tree code. And it's not like you simply can't allow multiple anonymous blockdevices for your networked filesystems similar to the one used for st_dev already.

But requiring 1500 anonymous blockdevices (== number of storage targets) be created at mount time, which exporting some varying-over-reboot, and inconsistent-across-clients random-value dev_t for network filesystems just for the possibility that the client is going to do FIEMAP isn't making the interface better either...

Getting devices of [0x1908afed, 0x4058204b] back from FIEMAP of a file on one client, and [0x4bac5821, 0x0abefd63] on another client is pretty useless compared to devices [2, 4], which have very clear meanings, will always be the same across all clients, and the same across reboots.

Quoted 11 lines
Quoted 6 lines

For RAID1/10 you can return multiple logical->physical extent mappings for the same logical range of the file with different "device" IDs. You could do the same for RAID5 returning each of the data and parity chunks with "NO_DIRECT" if desired (maybe only on the parity extent, or don't return the parity extent at all). The spec does not require that the returned extents be non-overlapping.

Umm, no. That's just make the interface too complicated. I can bet with your that userspace programmers will generally only test their code with simple filesystems and hell will break lose when they get these multiple ranges. Especially as that's a very unnatural interface.

The metadata information isn't exposed to callers by default, they have to request it explicitly with e.g. FIEMAP_FLAG_METADATA. For the most common use cases, applications/users will care about:
a) for cp/tar/dd/etc they only want to know where there are holes. This is available in the most simple instance of FIEMAP (no flags).
b) for "fiemap" the user will want to know whether there are large or small contiguous allocations/fragmentation, or just the extent count.
c) for sophisticated users (e.g. filesystem developers, performance tuning) they want to know both the extent information, the metadata layout, and possibly the mapping all the way down to the platters

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#19 lore raw
In reply to Christoph Hellwig

On Sun, May 25, 2008 at 03:42:03PM -0400, Christoph Hellwig wrote:

[ I'll get back to you regarding luns and the HSM flag ]

Quoted 8 lines
Quoted 6 lines

struct fiemap_extent_info {
unsigned int fi_flags; /* Flags as passed from user */
unsigned int fi_extents_mapped; /* Number of mapped extents */
unsigned int fi_extents_max; /* Size of fiemap_extent array */
char *fi_extents_start; /* Start of fiemap_extent array */
};

Why is this passes a structure instead of individual arguments?

[ The structure vs args seems to been addressed elsewhere in this thread ]

Quoted 1 line

Also why isn't fi_extents_start properly typed?

There's no good reason, my brain just wrote it that way. I can type it properly in my next patch.

Quoted 8 lines
Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

Just define a simple FIECOUNT ioctl.

Having it as a seperate, simpler ioctl is fine with me. As it is, we're sort of shoe-horning it into a structure which is more optimized for returning actual extent data. The flag breaks lots of rules for that structure, such as no actual fm_extents needing to be allocated, fm_extent_count is magically ignored for that call, etc. So the simpler API to userspace is a win, IMHO.

What about the back-end though? This is pretty transparently handled in fiemap_fill_next_extent() and many file systems (Ocfs2 included) would just have ->fiecount callbacks that are nearly identical ->fiecount to their ->fiemap...

	--Mark

--
Mark Fasheh

#20 lore raw
Quoted 7 lines · Mark Fasheh
Quoted 7 lines
Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

Quoted 6 lines
Quoted 1 line

Just define a simple FIECOUNT ioctl.

What about the back-end though? This is pretty transparently handled in fiemap_fill_next_extent() and many file systems (Ocfs2 included) would just have ->fiecount callbacks that are nearly identical ->fiecount to their ->fiemap...

Provide generic_fiemap_fiecount() that does the operation in
terms of ->fiemap(). Then filesystems like ocfs2 can just do .fiecount = generic_fiemap_fiecount.
I agree with Christoph that it seems a bit overloaded when done
as a special case of FIEMAP.

Joel

Signature · 12 lines
"This is the end, beautiful friend.
 This is the end, my only friend the end
 Of our elaborate plans, the end
 Of everything that stands, the end
 No safety or surprise, the end
 I'll never look into your eyes again."

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
#21 lore raw
Quoted 10 lines · Joel Becker
Quoted 9 lines · Mark Fasheh
Quoted 8 lines
Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

Just define a simple FIECOUNT ioctl.

Quoted 8 lines
Quoted 4 lines

What about the back-end though? This is pretty transparently handled in fiemap_fill_next_extent() and many file systems (Ocfs2 included) would just have ->fiecount callbacks that are nearly identical ->fiecount to their ->fiemap...

Provide generic_fiemap_fiecount() that does the operation in
terms of ->fiemap(). Then filesystems like ocfs2 can just do .fiecount = generic_fiemap_fiecount.

Right - that's essentially what I'm getting at. Some of Christophs comments later in this thread seemed geared towards implementation too, so I was just wondering...

	--Mark

--
Mark Fasheh

#22 lore raw
In reply to Joel Becker
Quoted 3 lines · Joel Becker

Provide generic_fiemap_fiecount() that does the operation in
terms of ->fiemap(). Then filesystems like ocfs2 can just do .fiecount = generic_fiemap_fiecount.

Yes, exactly. Although I suspect for most filesystems an iterator-based implementation like the one for ext4 in the patchset would actually be ceaner.

I like these iterators, they helped cleaning up a lot of messy interface in XFS.

#23 lore raw
In reply to Joel Becker
Quoted 15 lines · Joel Becker
Quoted 7 lines · Mark Fasheh
Quoted 7 lines
Quoted 3 lines

If the request has the FIEMAP_FLAG_NUM_EXTENTS flag set, then calling this helper is not necessary and fi_extents_mapped can be set directly.

Sounds like the count number of extents request should be a separate ioctl and separate filesystem entry point instead of overloading FIEMAP.

Quoted 6 lines
Quoted 1 line

Just define a simple FIECOUNT ioctl.

What about the back-end though? This is pretty transparently handled in fiemap_fill_next_extent() and many file systems (Ocfs2 included) would just have ->fiecount callbacks that are nearly identical ->fiecount to their ->fiemap...

Quoted 5 lines

Provide generic_fiemap_fiecount() that does the operation in
terms of ->fiemap(). Then filesystems like ocfs2 can just do .fiecount = generic_fiemap_fiecount.
I agree with Christoph that it seems a bit overloaded when done
as a special case of FIEMAP.

The question is whether there are any (or many) filesystems that will NOT implement ->fiecount() as a wrapper of ->fiemap()? At that point the "simplification" of the API means that there is actually more code and layers being called for the simple version.

I'm not saying I'm against having a FIECOUNT ioctl, we're probably just going to use an updated filefrag like everyone else, but it doesn't seem like a net reduction in code anywhere.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#24 lore raw
Quoted 7 lines · Andreas Dilger
Quoted 5 lines · Joel Becker

Provide generic_fiemap_fiecount() that does the operation in
terms of ->fiemap(). Then filesystems like ocfs2 can just do .fiecount = generic_fiemap_fiecount.
I agree with Christoph that it seems a bit overloaded when done
as a special case of FIEMAP.

Quoted 8 lines

The question is whether there are any (or many) filesystems that will NOT implement ->fiecount() as a wrapper of ->fiemap()? At that point the "simplification" of the API means that there is actually more code and layers being called for the simple version.

I'm not saying I'm against having a FIECOUNT ioctl, we're probably just going to use an updated filefrag like everyone else, but it doesn't seem like a net reduction in code anywhere.

It's not about net reduction of code. It's about a readable and
understandable interface. "Pass this array of extent structures we'll ignore if you set this special flag" is pretty ugly. Calling FIECOUNT separately is nice. How it is implemented in the kernel is a whole 'nother ball of wax - maybe we don't have ->fiecount() and always implement FIECOUNT in terms of a ->fiemap() walk. Doesn't matter.

Joel

Signature · 8 lines
"Hey mister if you're gonna walk on water,
 Could you drop a line my way?"

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
#25 lore raw
Quoted 17 lines · Joel Becker
Quoted 7 lines · Andreas Dilger
Quoted 5 lines · Joel Becker

Provide generic_fiemap_fiecount() that does the operation in
terms of ->fiemap(). Then filesystems like ocfs2 can just do .fiecount = generic_fiemap_fiecount.
I agree with Christoph that it seems a bit overloaded when done
as a special case of FIEMAP.

Quoted 8 lines

The question is whether there are any (or many) filesystems that will NOT implement ->fiecount() as a wrapper of ->fiemap()? At that point the "simplification" of the API means that there is actually more code and layers being called for the simple version.

I'm not saying I'm against having a FIECOUNT ioctl, we're probably just going to use an updated filefrag like everyone else, but it doesn't seem like a net reduction in code anywhere.

Quoted 6 lines

It's not about net reduction of code. It's about a readable and
understandable interface. "Pass this array of extent structures we'll ignore if you set this special flag" is pretty ugly. Calling FIECOUNT separately is nice. How it is implemented in the kernel is a whole 'nother ball of wax - maybe we don't have ->fiecount() and always implement FIECOUNT in terms of a ->fiemap() walk. Doesn't matter.

XFS has XFS_IOC_FSGETXATTR which can return the number of extents on an inode. It's a total count, not a range count, so it's a bit different to FIECOUNT and as such does not require walking the extent list to retrieve (extent count is in the inode itself).

It's still not that straight forward as you have to encode count, offset and length into a structure to pass into the ioctl. i.e. is it really that much simpler and cleaner than just adding a extra flag to FIEMAP?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#26 lore raw
Quoted 4 lines · Dave Chinner

XFS has XFS_IOC_FSGETXATTR which can return the number of extents on an inode. It's a total count, not a range count, so it's a bit different to FIECOUNT and as such does not require walking the extent list to retrieve (extent count is in the inode itself).

What use is there geeting the extent count for a range? I'd rather do it only per-file like the xfs ioctl.

Quoted 4 lines

It's still not that straight forward as you have to encode count, offset and length into a structure to pass into the ioctl. i.e. is it really that much simpler and cleaner than just adding a extra flag to FIEMAP?

Yes :)

#27 lore raw
Quoted 3 lines · Christoph Hellwig

What use is there geeting the extent count for a range? I'd rather do it only per-file like the xfs ioctl.

I'll answer that from practical experience. Our api equivalents:

__u64 fm_start; /* logical offset (inclusive) at
* which to start mapping (in) */
__u64 fm_length; /* logical length of mapping which
* userspace cares about (in) */
__u32 fm_extent_count; /* size of fm_extents array (in) */
__u32 fm_mapped_extents; /* number of extents that were
* mapped (out) */

... note it has no flags field and no separate ioctl_extent_count.

"fm_extent_count" is
IN == max_extents to return.
OUT == number of extents remaining in-range after fm_mapped_extents

Pass in fm_extent_count==0 and you get OUT number of extent entries within your fm_start + fm_length range. Which you can use to set your malloc size because the FS can have massive extent counts :(

This is why it was done. In practice this was only used by kernel callers because most application developers simply looped with a fixed buffer and adjusted fm_start. Dumber applications kept doubling their malloc to get all extents at once... or core dump :)

I'm not saying there is a good reason to do it this way in linux, just why someone else did it that way.

jim

#28 lore raw
Quoted 4 lines · jim owens

This is why it was done. In practice this was only used by kernel callers because most application developers simply looped with a fixed buffer and adjusted fm_start. Dumber applications kept doubling their malloc to get all extents at once... or core dump :)

I'm seeing that a database, VM, or filesystem-in-a-file app (anything like that using huge files) may want to optimise it's I/O scheduling and allocation pattern according to the estimated layout of it's open files... and some of those would like to be robust and not core dump when presented with really large files! :) And not core dumping does not mean "abandon the strategy when too large" either - it means have a strategy which scales :-)

I'm also seeing block devices potentially offering a similar or even same interface. Now that block devices (LVM) have extents and underlying allocation strategies etc. much the same as modern filesystems. Perhaps LVM itself could support FIEMAP. From a database engine's point of view, there isn't really much difference any more except differing APIs and limitations.

-- Jamie

#29 lore raw
In reply to jim owens
Quoted 17 lines · jim owens
Quoted 3 lines · Christoph Hellwig

What use is there geeting the extent count for a range? I'd rather do it only per-file like the xfs ioctl.

I'll answer that from practical experience. Our api equivalents:

__u64 fm_start; /* logical offset (inclusive) at
* which to start mapping (in) */
__u64 fm_length; /* logical length of mapping which
* userspace cares about (in) */
__u32 fm_extent_count; /* size of fm_extents array (in) */
__u32 fm_mapped_extents; /* number of extents that were
* mapped (out) */

... note it has no flags field and no separate ioctl_extent_count.

Quoted 7 lines

"fm_extent_count" is
IN == max_extents to return.
OUT == number of extents remaining in-range after fm_mapped_extents

Pass in fm_extent_count==0 and you get OUT number of extent entries within your fm_start + fm_length range. Which you can use to set your malloc size because the FS can have massive extent counts :(

See, that's a very natural API. I'd be much happier with that -
"extent_count == 0" is consistent with no extent structures passed in. It also fits with many other API that do it the same way (eg, snprintf).

Joel

Signature · 9 lines
"I don't know anything about music. In my line you don't have
 to."
        - Elvis Presley

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
#30 lore raw
Quoted 21 lines · Joel Becker
Quoted 20 lines · jim owens
Quoted 2 lines · Christoph Hellwig

What use is there geeting the extent count for a range? I'd rather do it only per-file like the xfs ioctl.

I'll answer that from practical experience. Our api equivalents:

__u64 fm_start; /* logical offset (inclusive) at
* which to start mapping (in) */
__u64 fm_length; /* logical length of mapping which
* userspace cares about (in) */
__u32 fm_extent_count; /* size of fm_extents array (in) */
__u32 fm_mapped_extents; /* number of extents that were
* mapped (out) */

... note it has no flags field and no separate ioctl_extent_count.

"fm_extent_count" is
IN == max_extents to return.
OUT == number of extents remaining in-range after fm_mapped_extents

Quoted 7 lines
Quoted 3 lines

Pass in fm_extent_count==0 and you get OUT number of extent entries within your fm_start + fm_length range. Which you can use to set your malloc size because the FS can have massive extent counts :(

See, that's a very natural API. I'd be much happier with that -
"extent_count == 0" is consistent with no extent structures passed in. It also fits with many other API that do it the same way (eg, snprintf).

So, to clarify, you are suggesting that FIEMAP_FLAG_NUM_EXTENTS isn't needed, and returning the extent count should just be detected by calling FIEMAP with fm_extent_count == 0? I'm OK with that also, since calling it with fm_extent_count == 0 doesn't make sense otherwise.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#31 lore raw
Quoted 4 lines · Andreas Dilger

So, to clarify, you are suggesting that FIEMAP_FLAG_NUM_EXTENTS isn't needed, and returning the extent count should just be detected by calling FIEMAP with fm_extent_count == 0? I'm OK with that also, since calling it with fm_extent_count == 0 doesn't make sense otherwise.

I'd always set fm_extent_count to whatever covers the range.
So, like snprintf(3), you always know when you were truncated and by how much. Then, passing fm_extent_count=0 in is just a special case of it. Like estimating a printf buffer with len = snprintf(NULL, 0, fmt, ...);

Joel

Signature · 9 lines
"Reader, suppose you were and idiot.  And suppose you were a member of
 Congress.  But I repeat myself."
	- Mark Twain

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
#32 lore raw
Quoted 10 lines · Joel Becker
Quoted 4 lines · Andreas Dilger

So, to clarify, you are suggesting that FIEMAP_FLAG_NUM_EXTENTS isn't needed, and returning the extent count should just be detected by calling FIEMAP with fm_extent_count == 0? I'm OK with that also, since calling it with fm_extent_count == 0 doesn't make sense otherwise.

I'd always set fm_extent_count to whatever covers the range.
So, like snprintf(3), you always know when you were truncated and by how much. Then, passing fm_extent_count=0 in is just a special case of it. Like estimating a printf buffer with len = snprintf(NULL, 0, fmt, ...);

I totally agree, I just didn't remember to include the start/end range in my email, though I think they are necessary.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#33 lore raw
In reply to Joel Becker
Quoted 6 lines · Joel Becker

It's not about net reduction of code. It's about a readable and
understandable interface. "Pass this array of extent structures we'll ignore if you set this special flag" is pretty ugly. Calling FIECOUNT separately is nice. How it is implemented in the kernel is a whole 'nother ball of wax - maybe we don't have ->fiecount() and always implement FIECOUNT in terms of a ->fiemap() walk. Doesn't matter.

In the FIEMAP + NUM_EXTENTS count, you don't need to pass the array of structures, just the header. I don't think this is an onerous interface:

	struct fiemap fiemap = { .fm_start = 0, .fm_length = ~0ULL,
				 .fm_num_extents = 0,
				 .fm_flags = FIEMAP_FLAG_NUM_EXTENTS }

	rc = ioctl(fd, FIEMAP, &fiemap);

	if (rc == 0)
		num_extents = fiemap.fm_mapped_extents;

We wouldn't need to even specify .fm_num_extents, if the VFS handler doesn't check its validity for FIEMAP_FLAG_NUM_EXTENTS (contrary to my recent change proposal).

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#34 lore raw
In reply to Andreas Dilger
Quoted 4 lines · Andreas Dilger

The question is whether there are any (or many) filesystems that will NOT implement ->fiecount() as a wrapper of ->fiemap()? At that point the "simplification" of the API means that there is actually more code and layers being called for the simple version.

ext4 for example as obvious from the code posted in this thread.