Skip to contentSkip to conversation

f2fs: change fiemap way in printing compression chunk

PATCH Superseded Daeho Jeong 3 participants 19 messages Jul 21, 2021 – Jul 22, 2021 1 message not in the archive F2FS file system

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 f2fs: change fiemap way in printing compression chunk

From: Daeho Jeong <[email protected]>

When we print out a discontinuous compression chunk, it shows like a continuous chunk now. To show it more correctly, I've changed the way of printing fiemap info like below. Plus, eliminated NEW_ADDR(-1) in fiemap info, since it is not in fiemap user api manual.

0: 0000000000000000 0000000000000000 0000000000001000 1008 (M/E) 1: 0000000000001000 0000000f15c0f000 0000000000001000 1008 (M/E) 2: 0000000000002000 0000000000000000 0000000000002000 1808 (M/U/E) 3: 0000000000004000 0000000000000000 0000000000001000 1008 (M/E) 4: 0000000000005000 0000000f15c10000 0000000000001000 1008 (M/E) 5: 0000000000006000 0000000000000000 0000000000002000 1808 (M/U/E) 6: 0000000000008000 0000000000000000 0000000000001000 1008 (M/E) M => FIEMAP_EXTENT_MERGED
E => FIEMAP_EXTENT_ENCODED
U => FIEMAP_EXTENT_UNWRITTEN

---
 fs/f2fs/data.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)
fs/f2fs/data.c +16 -26
index 3a01a1b50104..6e1be876d96b 100644
@@ -1845,6 +1845,7 @@ int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 	int ret = 0;
 	bool compr_cluster = false;
 	unsigned int cluster_size = F2FS_I(inode)->i_cluster_size;
+	unsigned int count_in_cluster;
 	loff_t maxbytes;
 
 	if (fieinfo->fi_flags & FIEMAP_FLAG_CACHE) {
@@ -1893,7 +1894,7 @@ int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 	map.m_seg_type = NO_CHECK_TYPE;
 
 	if (compr_cluster)
-		map.m_len = cluster_size - 1;
+		map.m_len = cluster_size - count_in_cluster;
 
 	ret = f2fs_map_blocks(inode, &map, 0, F2FS_GET_BLOCK_FIEMAP);
 	if (ret)
@@ -1926,37 +1927,26 @@ int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 	if (start_blk > last_blk)
 		goto out;
 
-	if (compr_cluster) {
-		compr_cluster = false;
-
-
-		logical = blks_to_bytes(inode, start_blk - 1);
-		phys = blks_to_bytes(inode, map.m_pblk);
-		size = blks_to_bytes(inode, cluster_size);
-
-		flags |= FIEMAP_EXTENT_ENCODED;
-
-		start_blk += cluster_size - 1;
-
-		if (start_blk > last_blk)
-			goto out;
-
-		goto prep_next;
-	}
-
-	if (map.m_pblk == COMPRESS_ADDR) {
-		compr_cluster = true;
-		start_blk++;
-		goto prep_next;
-	}
-
 	logical = blks_to_bytes(inode, start_blk);
-	phys = blks_to_bytes(inode, map.m_pblk);
+	phys = __is_valid_data_blkaddr(map.m_pblk) ?
+		blks_to_bytes(inode, map.m_pblk) : 0;
 	size = blks_to_bytes(inode, map.m_len);
 	flags = 0;
 	if (map.m_flags & F2FS_MAP_UNWRITTEN)
 		flags = FIEMAP_EXTENT_UNWRITTEN;
 
+	if (map.m_pblk == COMPRESS_ADDR) {
+		flags |= FIEMAP_EXTENT_ENCODED;
+		compr_cluster = true;
+		count_in_cluster = 1;
+	} else if (compr_cluster) {
+		flags |= FIEMAP_EXTENT_ENCODED;
+		if (map.m_len > 0)
+			count_in_cluster += map.m_len;
+		if (count_in_cluster == cluster_size)
+			compr_cluster = false;
+	}
+
 	start_blk += bytes_to_blks(inode, size);
 
 prep_next:
Signature · 8 lines
2.32.0.402.g57bb445576-goog



_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
#1 lore raw
Quoted 7 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

When we print out a discontinuous compression chunk, it shows like a continuous chunk now. To show it more correctly, I've changed the way of printing fiemap info like below. Plus, eliminated NEW_ADDR(-1) in fiemap info, since it is not in fiemap user api manual.

Quoted 7 lines

0: 0000000000000000 0000000000000000 0000000000001000 1008 (M/E) 1: 0000000000001000 0000000f15c0f000 0000000000001000 1008 (M/E) 2: 0000000000002000 0000000000000000 0000000000002000 1808 (M/U/E) 3: 0000000000004000 0000000000000000 0000000000001000 1008 (M/E) 4: 0000000000005000 0000000f15c10000 0000000000001000 1008 (M/E) 5: 0000000000006000 0000000000000000 0000000000002000 1808 (M/U/E) 6: 0000000000008000 0000000000000000 0000000000001000 1008 (M/E)

Please label these columns.

Anyway, this doesn't appear to work quite in the way I had in mind. With this patch, what I'm seeing is:

$ head -c 16384 /dev/zero > file; xfs_io -c "fiemap -v" file
file:
 EXT: FILE-OFFSET      BLOCK-RANGE      TOTAL FLAGS
   0: [0..7]:          0..7                 8 0x1008
   1: [8..15]:         2683128..2683135     8 0x1008
   2: [16..31]:        0..15               16 0x1809

So, working in 512-byte sectors, the logical sectors 0-31 are stored as one compressed cluster in the 8 physical sectors 2683128-2683135.

The problem is, with this patch these physical sectors are reported at logical sectors 8-15 instead of 0-7. Obviously, this isn't particularly well-defined, but I thought it was logical for the physical blocks to be associated with the first logical blocks. That is what the tests I've written (xfstest f2fs/002, and the Android vts_kernel_encryption_test) assume.

Is there any particular reason why you wouldn't report instead:

   0: [0..7]:         2683128..2683135     8 0x1008
   1: [8..31]:        0..23                8 0x1809

- Eric

#2 lore raw

Hi Eric,

Quoted 38 lines · Eric Biggers
Quoted 7 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

When we print out a discontinuous compression chunk, it shows like a continuous chunk now. To show it more correctly, I've changed the way of printing fiemap info like below. Plus, eliminated NEW_ADDR(-1) in fiemap info, since it is not in fiemap user api manual.

Quoted 7 lines

0: 0000000000000000 0000000000000000 0000000000001000 1008 (M/E) 1: 0000000000001000 0000000f15c0f000 0000000000001000 1008 (M/E) 2: 0000000000002000 0000000000000000 0000000000002000 1808 (M/U/E) 3: 0000000000004000 0000000000000000 0000000000001000 1008 (M/E) 4: 0000000000005000 0000000f15c10000 0000000000001000 1008 (M/E) 5: 0000000000006000 0000000000000000 0000000000002000 1808 (M/U/E) 6: 0000000000008000 0000000000000000 0000000000001000 1008 (M/E)

Please label these columns.

Anyway, this doesn't appear to work quite in the way I had in mind. With this patch, what I'm seeing is:

$ head -c 16384 /dev/zero > file; xfs_io -c "fiemap -v" file
file:
 EXT: FILE-OFFSET      BLOCK-RANGE      TOTAL FLAGS
   0: [0..7]:          0..7                 8 0x1008
   1: [8..15]:         2683128..2683135     8 0x1008
   2: [16..31]:        0..15               16 0x1809

So, working in 512-byte sectors, the logical sectors 0-31 are stored as one compressed cluster in the 8 physical sectors 2683128-2683135.

The problem is, with this patch these physical sectors are reported at logical sectors 8-15 instead of 0-7. Obviously, this isn't particularly well-defined, but I thought it was logical for the physical blocks to be associated with the first logical blocks. That is what the tests I've written (xfstest f2fs/002, and the Android vts_kernel_encryption_test) assume.

Quoted 6 lines

Is there any particular reason why you wouldn't report instead:

   0: [0..7]:         2683128..2683135     8 0x1008
   1: [8..31]:        0..23                8 0x1809

- Eric

The reason is related to how F2FS stores the mapping information in the mapping table.
Actually, the mapping inside of the table is like this. [0..7]: COMPR_ADDR flag(0x1008) -> merged, encoded [8..15]: 2683128..2683135 flag(0x1008) -> merged, encoded [16..31]: NEW_ADDR flag(0x1809) -> merged, unwritten(!), last_extent

I understand what you mean.
But, if we adapt to your way, how do you think we can print out when we ask for f2fs to print out only the [8..15] area? Zero address? How about flags?
I think the current way explains the layout of the f2fs metadata more exactly.

Thank you,

#3 lore raw
Quoted 55 lines · Daeho Jeong

Hi Eric,

Quoted 38 lines · Eric Biggers
Quoted 7 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

When we print out a discontinuous compression chunk, it shows like a continuous chunk now. To show it more correctly, I've changed the way of printing fiemap info like below. Plus, eliminated NEW_ADDR(-1) in fiemap info, since it is not in fiemap user api manual.

Quoted 7 lines

0: 0000000000000000 0000000000000000 0000000000001000 1008 (M/E) 1: 0000000000001000 0000000f15c0f000 0000000000001000 1008 (M/E) 2: 0000000000002000 0000000000000000 0000000000002000 1808 (M/U/E) 3: 0000000000004000 0000000000000000 0000000000001000 1008 (M/E) 4: 0000000000005000 0000000f15c10000 0000000000001000 1008 (M/E) 5: 0000000000006000 0000000000000000 0000000000002000 1808 (M/U/E) 6: 0000000000008000 0000000000000000 0000000000001000 1008 (M/E)

Please label these columns.

Anyway, this doesn't appear to work quite in the way I had in mind. With this patch, what I'm seeing is:

$ head -c 16384 /dev/zero > file; xfs_io -c "fiemap -v" file
file:
 EXT: FILE-OFFSET      BLOCK-RANGE      TOTAL FLAGS
   0: [0..7]:          0..7                 8 0x1008
   1: [8..15]:         2683128..2683135     8 0x1008
   2: [16..31]:        0..15               16 0x1809

So, working in 512-byte sectors, the logical sectors 0-31 are stored as one compressed cluster in the 8 physical sectors 2683128-2683135.

The problem is, with this patch these physical sectors are reported at logical sectors 8-15 instead of 0-7. Obviously, this isn't particularly well-defined, but I thought it was logical for the physical blocks to be associated with the first logical blocks. That is what the tests I've written (xfstest f2fs/002, and the Android vts_kernel_encryption_test) assume.

Quoted 6 lines

Is there any particular reason why you wouldn't report instead:

   0: [0..7]:         2683128..2683135     8 0x1008
   1: [8..31]:        0..23                8 0x1809

- Eric

The reason is related to how F2FS stores the mapping information in the mapping table.
Actually, the mapping inside of the table is like this. [0..7]: COMPR_ADDR flag(0x1008) -> merged, encoded [8..15]: 2683128..2683135 flag(0x1008) -> merged, encoded [16..31]: NEW_ADDR flag(0x1809) -> merged, unwritten(!), last_extent

Quoted 6 lines

I understand what you mean.
But, if we adapt to your way, how do you think we can print out when we ask for f2fs to print out only the [8..15] area? Zero address? How about flags?
I think the current way explains the layout of the f2fs metadata more exactly.

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

#4 lore raw
Quoted 74 lines · Eric Biggers
Quoted 55 lines · Daeho Jeong

Hi Eric,

Quoted 38 lines · Eric Biggers
Quoted 7 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

When we print out a discontinuous compression chunk, it shows like a continuous chunk now. To show it more correctly, I've changed the way of printing fiemap info like below. Plus, eliminated NEW_ADDR(-1) in fiemap info, since it is not in fiemap user api manual.

Quoted 7 lines

0: 0000000000000000 0000000000000000 0000000000001000 1008 (M/E) 1: 0000000000001000 0000000f15c0f000 0000000000001000 1008 (M/E) 2: 0000000000002000 0000000000000000 0000000000002000 1808 (M/U/E) 3: 0000000000004000 0000000000000000 0000000000001000 1008 (M/E) 4: 0000000000005000 0000000f15c10000 0000000000001000 1008 (M/E) 5: 0000000000006000 0000000000000000 0000000000002000 1808 (M/U/E) 6: 0000000000008000 0000000000000000 0000000000001000 1008 (M/E)

Please label these columns.

Anyway, this doesn't appear to work quite in the way I had in mind. With this patch, what I'm seeing is:

$ head -c 16384 /dev/zero > file; xfs_io -c "fiemap -v" file
file:
 EXT: FILE-OFFSET      BLOCK-RANGE      TOTAL FLAGS
   0: [0..7]:          0..7                 8 0x1008
   1: [8..15]:         2683128..2683135     8 0x1008
   2: [16..31]:        0..15               16 0x1809

So, working in 512-byte sectors, the logical sectors 0-31 are stored as one compressed cluster in the 8 physical sectors 2683128-2683135.

The problem is, with this patch these physical sectors are reported at logical sectors 8-15 instead of 0-7. Obviously, this isn't particularly well-defined, but I thought it was logical for the physical blocks to be associated with the first logical blocks. That is what the tests I've written (xfstest f2fs/002, and the Android vts_kernel_encryption_test) assume.

Quoted 6 lines

Is there any particular reason why you wouldn't report instead:

   0: [0..7]:         2683128..2683135     8 0x1008
   1: [8..31]:        0..23                8 0x1809

- Eric

The reason is related to how F2FS stores the mapping information in the mapping table.
Actually, the mapping inside of the table is like this. [0..7]: COMPR_ADDR flag(0x1008) -> merged, encoded [8..15]: 2683128..2683135 flag(0x1008) -> merged, encoded [16..31]: NEW_ADDR flag(0x1809) -> merged, unwritten(!), last_extent

Quoted 6 lines

I understand what you mean.
But, if we adapt to your way, how do you think we can print out when we ask for f2fs to print out only the [8..15] area? Zero address? How about flags?
I think the current way explains the layout of the f2fs metadata more exactly.

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Thank you,

#5 lore raw
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#6 lore raw
Quoted 44 lines · Eric Biggers
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

Quoted 8 lines

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

My last question is.
How about a discontinuous cluster like [0..31] maps to discontinuous three blocks like physical address 0x4, 0x14 and 0x24.
I think we have to return three extents for the one logical region like the below. What do you think?
[0..31] -> 0x4 (merged, encoded)
[0..31] -> 0x14 (merged, encoded)
[0..31] -> 0x24 (merged, encoded, last_extent)

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#7 lore raw
Quoted 54 lines · Daeho Jeong
Quoted 44 lines · Eric Biggers
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

Quoted 8 lines

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

Quoted 8 lines

My last question is.
How about a discontinuous cluster like [0..31] maps to discontinuous three blocks like physical address 0x4, 0x14 and 0x24.
I think we have to return three extents for the one logical region like the below. What do you think?
[0..31] -> 0x4 (merged, encoded)
[0..31] -> 0x14 (merged, encoded)
[0..31] -> 0x24 (merged, encoded, last_extent)

No, please don't do that. struct fiemap_extent only has a single length field, not separate lengths for fe_logical and fe_physical, so with your proposal there would be no way to know how many physical blocks to take from each extent. It would be reporting the same part of the file in contradictory ways.

Like I suggested originally, I think this case should be reported like:

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

It's not perfect, but I think it's the least bad option, for the reasons I've explained previously...

- Eric

#8 lore raw
Quoted 72 lines · Eric Biggers
Quoted 54 lines · Daeho Jeong
Quoted 44 lines · Eric Biggers
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

Quoted 8 lines

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

Quoted 8 lines

My last question is.
How about a discontinuous cluster like [0..31] maps to discontinuous three blocks like physical address 0x4, 0x14 and 0x24.
I think we have to return three extents for the one logical region like the below. What do you think?
[0..31] -> 0x4 (merged, encoded)
[0..31] -> 0x14 (merged, encoded)
[0..31] -> 0x24 (merged, encoded, last_extent)

No, please don't do that. struct fiemap_extent only has a single length field, not separate lengths for fe_logical and fe_physical, so with your proposal there would be no way to know how many physical blocks to take from each extent. It would be reporting the same part of the file in contradictory ways.

Like I suggested originally, I think this case should be reported like:

Quoted 8 lines
       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

It's not perfect, but I think it's the least bad option, for the reasons I've explained previously...

- Eric

Ok, I got your point. Let me try it again.

Thank you,

A message that is not in the archive (sent off-list or not yet received): CACOAw_yfG494AK=XH_xzeTDWn-a1mYF+537=VTT6oX6RgLGxnw@mail.gmail.com

#10 lore raw
Quoted 83 lines · Daeho Jeong
Quoted 70 lines · Eric Biggers
Quoted 54 lines · Daeho Jeong
Quoted 44 lines · Eric Biggers
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

Quoted 8 lines

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

Quoted 8 lines

My last question is.
How about a discontinuous cluster like [0..31] maps to discontinuous three blocks like physical address 0x4, 0x14 and 0x24.
I think we have to return three extents for the one logical region like the below. What do you think?
[0..31] -> 0x4 (merged, encoded)
[0..31] -> 0x14 (merged, encoded)
[0..31] -> 0x24 (merged, encoded, last_extent)

No, please don't do that. struct fiemap_extent only has a single length field, not separate lengths for fe_logical and fe_physical, so with your proposal there would be no way to know how many physical blocks to take from each extent. It would be reporting the same part of the file in contradictory ways.

Quoted 6 lines

Like I suggested originally, I think this case should be reported like:

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

Hi Eric,

I think we need to separate one more extent in the example to figure out that the 4 block cluster turned into 3 compressed blocks.

Quoted 7 lines
       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=4096
       fe_logical=12288 fe_physical=0 length=4096

What do you think?

[+linux-f2fs-devel, not sure why the list was dropped...]

I think that would work too, but what flags would you use in the last entry?

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#11 lore raw
Quoted 90 lines · Eric Biggers
Quoted 88 lines · Daeho Jeong
Quoted 70 lines · Eric Biggers
Quoted 54 lines · Daeho Jeong
Quoted 44 lines · Eric Biggers
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

Quoted 8 lines

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

Quoted 8 lines

My last question is.
How about a discontinuous cluster like [0..31] maps to discontinuous three blocks like physical address 0x4, 0x14 and 0x24.
I think we have to return three extents for the one logical region like the below. What do you think?
[0..31] -> 0x4 (merged, encoded)
[0..31] -> 0x14 (merged, encoded)
[0..31] -> 0x24 (merged, encoded, last_extent)

No, please don't do that. struct fiemap_extent only has a single length field, not separate lengths for fe_logical and fe_physical, so with your proposal there would be no way to know how many physical blocks to take from each extent. It would be reporting the same part of the file in contradictory ways.

Quoted 6 lines

Like I suggested originally, I think this case should be reported like:

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

Hi Eric,

I think we need to separate one more extent in the example to figure out that the 4 block cluster turned into 3 compressed blocks.

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=4096
       fe_logical=12288 fe_physical=0 length=4096
Quoted 8 lines
Quoted 2 lines

What do you think?

[+linux-f2fs-devel, not sure why the list was dropped...]

I think that would work too, but what flags would you use in the last entry?

- Eric

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#12 lore raw
Quoted 108 lines · Daeho Jeong
Quoted 90 lines · Eric Biggers
Quoted 88 lines · Daeho Jeong
Quoted 70 lines · Eric Biggers
Quoted 54 lines · Daeho Jeong
Quoted 44 lines · Eric Biggers
Quoted 20 lines · Daeho Jeong
Quoted 11 lines

How f2fs stores the mapping information doesn't matter. That's an implementation detail that shouldn't be exposed to userspace. The only thing that should be exposed is the actual mapping, and for that it seems natural to report the physical blocks first.

There is no perfect solution for how to handle the remaining logical blocks, given that the fiemap API was not designed for compressed files, but I think we should just go with extending the length of the last compressed extent in the cluster to cover the remaining logical blocks, i.e.:

Quoted 5 lines

[0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent

That's what btrfs does on compressed files.

- Eric

I also agree that that's an implementation detail that shouldn't be exposed to userspace.

Quoted 6 lines

I want to make it more clear for better appearance.

Do you think we have to remove "unwritten" information below? I also think it might be unnecessary information for the user. [0..31]: 2683128..2683159 flag(0x1009) -> merged, encoded, last_extent (unwritten?)

FIEMAP_EXTENT_UNWRITTEN already has a specific meaning; see Documentation/filesystems/fiemap.rst. It means that the data is all zeroes, and the disk space is preallocated but the data hasn't been written to disk yet.

In this case, the data is *not* necessarily all zeroes. So I think FIEMAP_EXTENT_UNWRITTEN shouldn't be used here.

Quoted 4 lines

Do you want f2fs to print out the info on a cluster basis, even when the user asks for one block information?
Like
If the user asks for the info of [8..15], f2fs will return the info of [0..31]?

Yes, since that's how FS_IOC_FIEMAP is supposed to work; see Documentation/filesystems/fiemap.rst:

Quoted 8 lines

All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request.

(That being said, the f2fs compression+encryption tests I've written don't exercise this case; they only map the whole file at once.)

- Eric

Quoted 8 lines

My last question is.
How about a discontinuous cluster like [0..31] maps to discontinuous three blocks like physical address 0x4, 0x14 and 0x24.
I think we have to return three extents for the one logical region like the below. What do you think?
[0..31] -> 0x4 (merged, encoded)
[0..31] -> 0x14 (merged, encoded)
[0..31] -> 0x24 (merged, encoded, last_extent)

No, please don't do that. struct fiemap_extent only has a single length field, not separate lengths for fe_logical and fe_physical, so with your proposal there would be no way to know how many physical blocks to take from each extent. It would be reporting the same part of the file in contradictory ways.

Quoted 6 lines

Like I suggested originally, I think this case should be reported like:

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

Hi Eric,

I think we need to separate one more extent in the example to figure out that the 4 block cluster turned into 3 compressed blocks.

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=4096
       fe_logical=12288 fe_physical=0 length=4096
Quoted 8 lines
Quoted 2 lines

What do you think?

[+linux-f2fs-devel, not sure why the list was dropped...]

I think that would work too, but what flags would you use in the last entry?

- Eric

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096
Quoted 4 lines

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

Well, as I said before, using UNWRITTEN isn't appropriate because it indicates that the data is all zeroes, which in this case it is not. Similarly, reporting a hole isn't appropriate because it also indicates that the data is all zeroes and also that it has no space allocated on-disk at all.

I think we should just over-report the physical length of the last extent in the cluster, which is what btrfs does...

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#13 lore raw
Quoted 13 lines · Eric Biggers
Quoted 8 lines

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096
Quoted 4 lines

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

Quoted 8 lines

Well, as I said before, using UNWRITTEN isn't appropriate because it indicates that the data is all zeroes, which in this case it is not. Similarly, reporting a hole isn't appropriate because it also indicates that the data is all zeroes and also that it has no space allocated on-disk at all.

I think we should just over-report the physical length of the last extent in the cluster, which is what btrfs does...

Also keep in mind that as far as fiemap is concerned, when FIEMAP_EXTENT_ENCODED is set (indicating that the data is compressed or encrypted), then reading the data from disk will have "undefined results"; see Documentation/filesystems/fiemap.rst. As such, when someone decides to do so anyway (which is necessary for encryption testing), they really need to know *exactly* what they're doing. So I think it's less bad to bend the rules on extents where FIEMAP_EXTENT_ENCODED is already set.

In contrast, your suggestion would incorreectly report some parts of the file as standard extents (or holes) without FIEMAP_EXTENT_ENCODED, so it would be expected that the standard meaning would apply to those parts.

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#14 lore raw
Quoted 32 lines · Eric Biggers
Quoted 13 lines · Eric Biggers
Quoted 8 lines

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096
Quoted 4 lines

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

Quoted 8 lines

Well, as I said before, using UNWRITTEN isn't appropriate because it indicates that the data is all zeroes, which in this case it is not. Similarly, reporting a hole isn't appropriate because it also indicates that the data is all zeroes and also that it has no space allocated on-disk at all.

I think we should just over-report the physical length of the last extent in the cluster, which is what btrfs does...

Also keep in mind that as far as fiemap is concerned, when FIEMAP_EXTENT_ENCODED is set (indicating that the data is compressed or encrypted), then reading the data from disk will have "undefined results"; see Documentation/filesystems/fiemap.rst. As such, when someone decides to do so anyway (which is necessary for encryption testing), they really need to know *exactly* what they're doing. So I think it's less bad to bend the rules on extents where FIEMAP_EXTENT_ENCODED is already set.

Quoted 5 lines

In contrast, your suggestion would incorreectly report some parts of the file as standard extents (or holes) without FIEMAP_EXTENT_ENCODED, so it would be expected that the standard meaning would apply to those parts.

- Eric

I also think it is okay with the FIEMAP_EXTENT_ENCODED flag.
It is actually all zero in a view of the filesystem internal and still undefined if read directly from the device.
If we remove this extent, it might be confusing to understand the layout of it.

Plus, I think we need to remove the last extent, when we return back the block to the free space pool to filesystem using releasing reserved space ioctl.

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#15 lore raw
Quoted 37 lines · Daeho Jeong
Quoted 36 lines · Eric Biggers
Quoted 13 lines · Eric Biggers
Quoted 8 lines

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096
Quoted 4 lines

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

Quoted 8 lines

Well, as I said before, using UNWRITTEN isn't appropriate because it indicates that the data is all zeroes, which in this case it is not. Similarly, reporting a hole isn't appropriate because it also indicates that the data is all zeroes and also that it has no space allocated on-disk at all.

I think we should just over-report the physical length of the last extent in the cluster, which is what btrfs does...

Also keep in mind that as far as fiemap is concerned, when FIEMAP_EXTENT_ENCODED is set (indicating that the data is compressed or encrypted), then reading the data from disk will have "undefined results"; see Documentation/filesystems/fiemap.rst. As such, when someone decides to do so anyway (which is necessary for encryption testing), they really need to know *exactly* what they're doing. So I think it's less bad to bend the rules on extents where FIEMAP_EXTENT_ENCODED is already set.

In contrast, your suggestion would incorreectly report some parts of the file as standard extents (or holes) without FIEMAP_EXTENT_ENCODED, so it would be expected that the standard meaning would apply to those parts.

Quoted 6 lines
Quoted 1 line

- Eric

I also think it is okay with the FIEMAP_EXTENT_ENCODED flag.
It is actually all zero in a view of the filesystem internal and still undefined if read directly from the device.
If we remove this extent, it might be confusing to understand the layout of it.

FIEMAP_EXTENT_ENCODED | FIEMAP_EXTENT_UNWRITTEN is contradictory, so that doesn't seem like a good option. And no, the range is *not* zero when read from the file by userspace, which is what UNWRITTEN is supposed to indicate.

Compressed data is terminated by an end-of-stream marker, so it is possible to decompress even if extra data is appended to it. So that's another reason why I feel that my suggestion is not as bad as the other options.

Quoted 3 lines

Plus, I think we need to remove the last extent, when we return back the block to the free space pool to filesystem using releasing reserved space ioctl.

This seems to be a filesystem implementation detail. Again, FS_IOC_FIEMAP is just about returning the actual extent mapping, not about other filesystem implementation details.

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#16 lore raw
Quoted 58 lines · Eric Biggers
Quoted 37 lines · Daeho Jeong
Quoted 36 lines · Eric Biggers
Quoted 13 lines · Eric Biggers
Quoted 8 lines

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096
Quoted 4 lines

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

Quoted 8 lines

Well, as I said before, using UNWRITTEN isn't appropriate because it indicates that the data is all zeroes, which in this case it is not. Similarly, reporting a hole isn't appropriate because it also indicates that the data is all zeroes and also that it has no space allocated on-disk at all.

I think we should just over-report the physical length of the last extent in the cluster, which is what btrfs does...

Also keep in mind that as far as fiemap is concerned, when FIEMAP_EXTENT_ENCODED is set (indicating that the data is compressed or encrypted), then reading the data from disk will have "undefined results"; see Documentation/filesystems/fiemap.rst. As such, when someone decides to do so anyway (which is necessary for encryption testing), they really need to know *exactly* what they're doing. So I think it's less bad to bend the rules on extents where FIEMAP_EXTENT_ENCODED is already set.

In contrast, your suggestion would incorreectly report some parts of the file as standard extents (or holes) without FIEMAP_EXTENT_ENCODED, so it would be expected that the standard meaning would apply to those parts.

Quoted 6 lines
Quoted 1 line

- Eric

I also think it is okay with the FIEMAP_EXTENT_ENCODED flag.
It is actually all zero in a view of the filesystem internal and still undefined if read directly from the device.
If we remove this extent, it might be confusing to understand the layout of it.

FIEMAP_EXTENT_ENCODED | FIEMAP_EXTENT_UNWRITTEN is contradictory, so that doesn't seem like a good option. And no, the range is *not* zero when read from the file by userspace, which is what UNWRITTEN is supposed to indicate.

Compressed data is terminated by an end-of-stream marker, so it is possible to decompress even if extra data is appended to it. So that's another reason why I feel that my suggestion is not as bad as the other options.

Quoted 3 lines

Plus, I think we need to remove the last extent, when we return back the block to the free space pool to filesystem using releasing reserved space ioctl.

Quoted 5 lines

This seems to be a filesystem implementation detail. Again, FS_IOC_FIEMAP is just about returning the actual extent mapping, not about other filesystem implementation details.

- Eric

Understood. For confirmation, is this your final suggestion?

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#17 lore raw
Quoted 63 lines · Daeho Jeong
Quoted 62 lines · Eric Biggers
Quoted 37 lines · Daeho Jeong
Quoted 36 lines · Eric Biggers
Quoted 13 lines · Eric Biggers
Quoted 8 lines

I think it matches with FIEMAP_EXTENT_UNWRITTEN. Otherwise, we should shorten the last extent like below.

        fe_logical=0    fe_physical=16384  length=4096
        fe_logical=4096 fe_physical=81920  length=4096
        fe_logical=8192 fe_physical=147456 length=4096
Quoted 4 lines

Unwritten extent - the extent is allocated but its data has not been initialized. This indicates the extent's data will be all zero if read through the filesystem but the contents are undefined if read directly from the device.

Quoted 8 lines

Well, as I said before, using UNWRITTEN isn't appropriate because it indicates that the data is all zeroes, which in this case it is not. Similarly, reporting a hole isn't appropriate because it also indicates that the data is all zeroes and also that it has no space allocated on-disk at all.

I think we should just over-report the physical length of the last extent in the cluster, which is what btrfs does...

Also keep in mind that as far as fiemap is concerned, when FIEMAP_EXTENT_ENCODED is set (indicating that the data is compressed or encrypted), then reading the data from disk will have "undefined results"; see Documentation/filesystems/fiemap.rst. As such, when someone decides to do so anyway (which is necessary for encryption testing), they really need to know *exactly* what they're doing. So I think it's less bad to bend the rules on extents where FIEMAP_EXTENT_ENCODED is already set.

In contrast, your suggestion would incorreectly report some parts of the file as standard extents (or holes) without FIEMAP_EXTENT_ENCODED, so it would be expected that the standard meaning would apply to those parts.

Quoted 6 lines
Quoted 1 line

- Eric

I also think it is okay with the FIEMAP_EXTENT_ENCODED flag.
It is actually all zero in a view of the filesystem internal and still undefined if read directly from the device.
If we remove this extent, it might be confusing to understand the layout of it.

FIEMAP_EXTENT_ENCODED | FIEMAP_EXTENT_UNWRITTEN is contradictory, so that doesn't seem like a good option. And no, the range is *not* zero when read from the file by userspace, which is what UNWRITTEN is supposed to indicate.

Compressed data is terminated by an end-of-stream marker, so it is possible to decompress even if extra data is appended to it. So that's another reason why I feel that my suggestion is not as bad as the other options.

Quoted 3 lines

Plus, I think we need to remove the last extent, when we return back the block to the free space pool to filesystem using releasing reserved space ioctl.

This seems to be a filesystem implementation detail. Again, FS_IOC_FIEMAP is just about returning the actual extent mapping, not about other filesystem implementation details.

Quoted 7 lines
Quoted 1 line

- Eric

Understood. For confirmation, is this your final suggestion?

       fe_logical=0    fe_physical=16384  length=4096
       fe_logical=4096 fe_physical=81920  length=4096
       fe_logical=8192 fe_physical=147456 length=8192

Yes.

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#18 lore raw
In reply to Daeho Jeong

Btw, any chane you could switch f2fs to use iomap_fiemap instead of the handrolled version? Especially with the work from Eric to add iomap based direct I/O this should be much simpler now.

#19 lore raw

Good point. I already made my second version of this. We could revisit later to make it simple.

Thank you,

Quoted 4 lines · Christoph Hellwig

Btw, any chane you could switch f2fs to use iomap_fiemap instead of the handrolled version? Especially with the work from Eric to add iomap based direct I/O this should be much simpler now.