Skip to contentSkip to conversation

when does btrfs create sparse extents?

DISCUSSION Marek Behun 4 participants 14 messages Apr 22, 2020 – Apr 23, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,

there was a bug fixed recently in U-Boot's btrfs driver - the driver failed to read files with sparse extents. This causes that sometimes device failes to boot Linux, since the kernel fails to load from storage.

So when does kernel's btrfs driver write sparse extents? Is it always when it finds a PAGE_SIZEd and aligned all-zeros block? Or is it more complicated?

Thanks.

Marek

#1 lore raw
Quoted 11 lines · Marek Behun

Hello,

there was a bug fixed recently in U-Boot's btrfs driver - the driver failed to read files with sparse extents. This causes that sometimes device failes to boot Linux, since the kernel fails to load from storage.

So when does kernel's btrfs driver write sparse extents? Is it always when it finds a PAGE_SIZEd and aligned all-zeros block? Or is it more complicated?

I'm not a btrfs developer, so with respect to kernel code behavior I can't answer directly.

But I wonder if other sources of this sparseness has been considered? Maybe the build system is creating or preserving sparseness? e.g. tar --hole-detection or --sparse is used.

Another possibility is Btrfs supports two kinds of holes in the on-disk format for sparse files. Maybe uboot only supported the original (current default) type, and the bug really fixed the newer 'no-holes' feature version?

Signature · 1 line
Chris Murphy
#2 lore raw
Quoted 3 lines · On Wed, 22 Apr 2020 14:26:31 -0600 Chris Murphy <[email protected]> wrote:

But I wonder if other sources of this sparseness has been considered? Maybe the build system is creating or preserving sparseness? e.g. tar --hole-detection or --sparse is used.

I don't actually know how the kernel is copied into the /boot directory in this case.

Quoted 4 lines

Another possibility is Btrfs supports two kinds of holes in the on-disk format for sparse files. Maybe uboot only supported the original (current default) type, and the bug really fixed the newer 'no-holes' feature version?

No, U-Boot did not support holes at all.

#3 lore raw
In reply to Chris Murphy

e.g. from a 10m file created with truncate on two Btrfs file systems

original holes format (default)

item 6 key (257 EXTENT_DATA 0) itemoff 15768 itemsize 53
generation 7412 type 1 (regular)
extent data disk byte 0 nr 0
extent data offset 0 nr 10485760 ram 10485760 extent compression 0 (none)

On a file system with no-holes feature set, this item simply doesn't exist. I think basically it works by inference. Both kinds of files have size in the INODE_ITEM, e.g.

    item 4 key (257 INODE_ITEM 0) itemoff 32245 itemsize 160
        generation 889509 transid 889509 size 10485760 nbytes 0

Sparse extents are explicitly stated in the original format with disk byte 0 in an EXTENT_DATA item; whereas in the newer format, sparse extents exist whenever EXTENT_DATA items don't completely describe the file's size.

--
Chris Murphy

#4 lore raw
Quoted 14 lines · On Wed, 22 Apr 2020 14:44:46 -0600 Chris Murphy <[email protected]> wrote:

e.g. from a 10m file created with truncate on two Btrfs file systems

original holes format (default)

item 6 key (257 EXTENT_DATA 0) itemoff 15768 itemsize 53
generation 7412 type 1 (regular)
extent data disk byte 0 nr 0
extent data offset 0 nr 10485760 ram 10485760 extent compression 0 (none)

On a file system with no-holes feature set, this item simply doesn't exist. I think basically it works by inference. Both kinds of files have size in the INODE_ITEM, e.g.

Quoted 7 lines
    item 4 key (257 INODE_ITEM 0) itemoff 32245 itemsize 160
        generation 889509 transid 889509 size 10485760 nbytes 0

Sparse extents are explicitly stated in the original format with disk byte 0 in an EXTENT_DATA item; whereas in the newer format, sparse extents exist whenever EXTENT_DATA items don't completely describe the file's size.

Ok this means that U-Boot currently gained support for the original sparse extents.

I fear that current u-boot does not handle the new no-holes feature.

#5 lore raw
Quoted 26 lines · Marek Behun
Quoted 14 lines · On Wed, 22 Apr 2020 14:44:46 -0600 Chris Murphy <[email protected]> wrote:

e.g. from a 10m file created with truncate on two Btrfs file systems

original holes format (default)

item 6 key (257 EXTENT_DATA 0) itemoff 15768 itemsize 53
generation 7412 type 1 (regular)
extent data disk byte 0 nr 0
extent data offset 0 nr 10485760 ram 10485760 extent compression 0 (none)

On a file system with no-holes feature set, this item simply doesn't exist. I think basically it works by inference. Both kinds of files have size in the INODE_ITEM, e.g.

Quoted 7 lines
    item 4 key (257 INODE_ITEM 0) itemoff 32245 itemsize 160
        generation 889509 transid 889509 size 10485760 nbytes 0

Sparse extents are explicitly stated in the original format with disk byte 0 in an EXTENT_DATA item; whereas in the newer format, sparse extents exist whenever EXTENT_DATA items don't completely describe the file's size.

Quoted 4 lines

Ok this means that U-Boot currently gained support for the original sparse extents.

I fear that current u-boot does not handle the new no-holes feature.

I'd advise it should support it, I think it's expected to become the default. But I don't know the time frame for that.

Also, when the no-holes feature flag is not set, you can be certain only the original hole type is used. But when no-holes feature is set, it's possible for both types of holes to exist, because no-holes can be set after mkfs time by btrfstune.

Signature · 1 line
Chris Murphy
#12 lore raw
In reply to Marek Behun

22.04.2020 21:52, Marek Behun пишет:

Quoted 7 lines

Hello,

there was a bug fixed recently in U-Boot's btrfs driver - the driver failed to read files with sparse extents. This causes that sometimes device failes to boot Linux, since the kernel fails to load from storage.

Do you mean that kernel image (vmlinuz?) had holes? Or there were some other files that caused U-Boot to fail?

Quoted 8 lines

So when does kernel's btrfs driver write sparse extents? Is it always when it finds a PAGE_SIZEd and aligned all-zeros block? Or is it more complicated?

Thanks.

Marek

#13 lore raw
Quoted 11 lines · On Thu, 23 Apr 2020 08:57:59 +0300 Andrei Borzenkov <[email protected]> wrote:

22.04.2020 21:52, Marek Behun пишет:

Quoted 7 lines

Hello,

there was a bug fixed recently in U-Boot's btrfs driver - the driver failed to read files with sparse extents. This causes that sometimes device failes to boot Linux, since the kernel fails to load from storage.

Do you mean that kernel image (vmlinuz?) had holes? Or there were some other files that caused U-Boot to fail?

Kernel image (non-compressed Image for arm64) had ~200 PAGE_SIZEd all-zero blocks.

#6 lore raw
In reply to Marek Behun
Quoted 21 lines · Marek Behun
Quoted 17 lines · On Wed, 22 Apr 2020 14:44:46 -0600 Chris Murphy <[email protected]> wrote:

e.g. from a 10m file created with truncate on two Btrfs file systems

original holes format (default)

item 6 key (257 EXTENT_DATA 0) itemoff 15768 itemsize 53
generation 7412 type 1 (regular)
extent data disk byte 0 nr 0
extent data offset 0 nr 10485760 ram 10485760 extent compression 0 (none)

On a file system with no-holes feature set, this item simply doesn't exist. I think basically it works by inference. Both kinds of files have size in the INODE_ITEM, e.g.

    item 4 key (257 INODE_ITEM 0) itemoff 32245 itemsize 160
        generation 889509 transid 889509 size 10485760 nbytes 0
Quoted 7 lines
Quoted 4 lines

Sparse extents are explicitly stated in the original format with disk byte 0 in an EXTENT_DATA item; whereas in the newer format, sparse extents exist whenever EXTENT_DATA items don't completely describe the file's size.

Ok this means that U-Boot currently gained support for the original sparse extents.

To clear any confusion, what you mean by sparse extents is actually holes. The concept of sparse files exists (files with holes, regions of a file for which there is no allocated extent), but not sparse extents.

Quoted 2 lines

I fear that current u-boot does not handle the new no-holes feature.

The no-holes feature has been around since 2013, not exactly new, but it's not the default yet when creating a new filesystem.

As it has been mentioned earlier by Chris, it just removes the need for explicitly having metadata representing holes.
When not using the no-holes feature, there is an explicit file extent item pointing to a disk location of 0 (disk_bytenr field has a value of 0) for each file hole.
When using no-holes, there's no such file extent item - btrfs knows about the hole by checking that there is a gap between two consecutive file extent items (both having a disk_bytenr > 0).

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#7 lore raw
Quoted 40 lines · On Thu, 23 Apr 2020 11:49:16 +0100 Filipe Manana <[email protected]> wrote:
Quoted 21 lines · Marek Behun
Quoted 17 lines · On Wed, 22 Apr 2020 14:44:46 -0600 Chris Murphy <[email protected]> wrote:

e.g. from a 10m file created with truncate on two Btrfs file systems

original holes format (default)

item 6 key (257 EXTENT_DATA 0) itemoff 15768 itemsize 53
generation 7412 type 1 (regular)
extent data disk byte 0 nr 0
extent data offset 0 nr 10485760 ram 10485760 extent compression 0 (none)

On a file system with no-holes feature set, this item simply doesn't exist. I think basically it works by inference. Both kinds of files have size in the INODE_ITEM, e.g.

    item 4 key (257 INODE_ITEM 0) itemoff 32245 itemsize 160
        generation 889509 transid 889509 size 10485760 nbytes 0
Quoted 7 lines
Quoted 4 lines

Sparse extents are explicitly stated in the original format with disk byte 0 in an EXTENT_DATA item; whereas in the newer format, sparse extents exist whenever EXTENT_DATA items don't completely describe the file's size.

Ok this means that U-Boot currently gained support for the original sparse extents.

To clear any confusion, what you mean by sparse extents is actually holes. The concept of sparse files exists (files with holes, regions of a file for which there is no allocated extent), but not sparse extents.

Quoted 2 lines

I fear that current u-boot does not handle the new no-holes feature.

The no-holes feature has been around since 2013, not exactly new, but it's not the default yet when creating a new filesystem.

Quoted 8 lines

As it has been mentioned earlier by Chris, it just removes the need for explicitly having metadata representing holes.
When not using the no-holes feature, there is an explicit file extent item pointing to a disk location of 0 (disk_bytenr field has a value of 0) for each file hole.
When using no-holes, there's no such file extent item - btrfs knows about the hole by checking that there is a gap between two consecutive file extent items (both having a disk_bytenr > 0).

This I already understand. My main question though is: does kernel or btrfs do checking (at least sometimes) when writing a block of data onto disk if this block is all zero, and if yes, then this block is written as a hole (either by writing hole item or not writing anything)?

Or does this happen ONLY when requested by userspace?

Because for the love of god I cannot find why our kernel is being written this way onto disk - the installer doesn't explicitly request for PUNCH_HOLES nor anything, as far as I looked.

Marek

Marek

#8 lore raw
Quoted 53 lines · Marek Behun
Quoted 40 lines · On Thu, 23 Apr 2020 11:49:16 +0100 Filipe Manana <[email protected]> wrote:
Quoted 21 lines · Marek Behun
Quoted 17 lines · On Wed, 22 Apr 2020 14:44:46 -0600 Chris Murphy <[email protected]> wrote:

e.g. from a 10m file created with truncate on two Btrfs file systems

original holes format (default)

item 6 key (257 EXTENT_DATA 0) itemoff 15768 itemsize 53
generation 7412 type 1 (regular)
extent data disk byte 0 nr 0
extent data offset 0 nr 10485760 ram 10485760 extent compression 0 (none)

On a file system with no-holes feature set, this item simply doesn't exist. I think basically it works by inference. Both kinds of files have size in the INODE_ITEM, e.g.

    item 4 key (257 INODE_ITEM 0) itemoff 32245 itemsize 160
        generation 889509 transid 889509 size 10485760 nbytes 0
Quoted 7 lines
Quoted 4 lines

Sparse extents are explicitly stated in the original format with disk byte 0 in an EXTENT_DATA item; whereas in the newer format, sparse extents exist whenever EXTENT_DATA items don't completely describe the file's size.

Ok this means that U-Boot currently gained support for the original sparse extents.

To clear any confusion, what you mean by sparse extents is actually holes. The concept of sparse files exists (files with holes, regions of a file for which there is no allocated extent), but not sparse extents.

Quoted 2 lines

I fear that current u-boot does not handle the new no-holes feature.

The no-holes feature has been around since 2013, not exactly new, but it's not the default yet when creating a new filesystem.

Quoted 8 lines

As it has been mentioned earlier by Chris, it just removes the need for explicitly having metadata representing holes.
When not using the no-holes feature, there is an explicit file extent item pointing to a disk location of 0 (disk_bytenr field has a value of 0) for each file hole.
When using no-holes, there's no such file extent item - btrfs knows about the hole by checking that there is a gap between two consecutive file extent items (both having a disk_bytenr > 0).

Quoted 6 lines

This I already understand. My main question though is: does kernel or btrfs do checking (at least sometimes) when writing a block of data onto disk if this block is all zero, and if yes, then this block is written as a hole (either by writing hole item or not writing anything)?

Or does this happen ONLY when requested by userspace?

There's nothing in btrfs that converts a sequence of zeroes automatically to a hole.

It always has to be done by user space, either by writes that leave holes intentionally (e.g. create file, write 64K to offset 0, write 4K to offset 128, leaves a hole from range 64K to 128K) or by hole punching through fallocate().

Quoted 8 lines

Because for the love of god I cannot find why our kernel is being written this way onto disk - the installer doesn't explicitly request for PUNCH_HOLES nor anything, as far as I looked.

Marek

Marek

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#9 lore raw
Quoted 7 lines · On Thu, 23 Apr 2020 12:51:32 +0100 Filipe Manana <[email protected]> wrote:

There's nothing in btrfs that converts a sequence of zeroes automatically to a hole.

It always has to be done by user space, either by writes that leave holes intentionally (e.g. create file, write 64K to offset 0, write 4K to offset 128, leaves a hole from range 64K to 128K) or by hole punching through fallocate().

Thanks for this information. If you ever come to Prague, let me know, we can have a beer :D

#10 lore raw
Quoted 7 lines · Marek Behun
Quoted 3 lines · On Thu, 23 Apr 2020 12:51:32 +0100 Filipe Manana <[email protected]> wrote:

There's nothing in btrfs that converts a sequence of zeroes automatically to a hole.

Quoted 7 lines
Quoted 4 lines

It always has to be done by user space, either by writes that leave holes intentionally (e.g. create file, write 64K to offset 0, write 4K to offset 128, leaves a hole from range 64K to 128K) or by hole punching through fallocate().

Thanks for this information. If you ever come to Prague, let me know, we can have a beer :D

Noted! (Through it will be a long while until travel is allowed and safe)

Another case I forgot is a hole created by truncation - truncate a file to a larger size, then you get a hole for the range that goes from the old file size to the new file size.

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#11 lore raw
Quoted 14 lines · Filipe Manana
Quoted 12 lines · Marek Behun
Quoted 7 lines · On Thu, 23 Apr 2020 12:51:32 +0100 Filipe Manana <[email protected]> wrote:

There's nothing in btrfs that converts a sequence of zeroes automatically to a hole.

It always has to be done by user space, either by writes that leave holes intentionally (e.g. create file, write 64K to offset 0, write 4K to offset 128, leaves a hole from range 64K to 128K) or by hole punching through fallocate().

Quoted 8 lines
Quoted 2 lines

Thanks for this information. If you ever come to Prague, let me know, we can have a beer :D

Noted! (Through it will be a long while until travel is allowed and safe)

Another case I forgot is a hole created by truncation - truncate a file to a larger size, then you get a hole for the range that goes from the old file size to the new file size.

I recently discovered the benefit of using both truncate and partially fallocating a VM backing file.

I start with truncate so I can set a typical device size for the guest's view, which I don't have the local space if I were to fallocate. e.g. 100G local, but do truncate -s 1T. Then follow that with fallocate -l 12G so that I don't end up with a massively fragmented backing file.

To be clear, this backing file is set with chattr +C and never snapshot or reflinked. Btrfs initially doesn't seem to perform any different with sparse or fallocated backing files. But with aging, the sparse file can get really fragmented depending on the guest file system. Tens of thousands of extents are possible, and that does impact performance it seems.

Signature · 1 line
Chris Murphy