Skip to contentSkip to conversation

newer /bin/cp have worse btrfs fiemap performance.

DISCUSSION Wang Yugui 2 participants 3 messages Nov 5, 2022 – Nov 7, 2022 Btrfs file system

lore
Share & tools
lore

Wang Yugui

author
#0 lore raw

Hi,

newer /bin/cp have worse btrfs fiemap performance.

btrfs version: misc-next 750de989d367(the lastest of 2022/11/05) /bin/cp versions(run them on the same server)

    8.22 copy from centos/7.9
    8.30 rocklinux/9.0
    9.1 local build, https://kojipkgs.fedoraproject.org/packages/coreutils/9.1/8.eln121/

test case:
/bin/cp /mnt/test/file1 /dev/null
/mnt/test/file1 is created by https://lore.kernel.org/linux-btrfs/[email protected]/T/#T
and /mnt/test/file1 is 256M.

file is not cached: 'echo 3 >/proc/sys/vm/drop_caches' file is cached: run '/bin/cp /mnt/test/file1 /dev/null' again.

performance result(/bin/cp /mnt/test/file1 /dev/null):
/bin/cp 9.1

    file is not cached: 94.85(1:34.85)
    file is cached: 1982.43(33:02.43)
/bin/cp 8.30
    file is not cached: 1.48(0:01.48)
    file is cached:14.07(0:14.07)
/bin/cp 8.22
    file is not cached: 0.53(0:00.53)
    file is cached: 0.10(0:00.10)

as a compare, we test it on xfs too.
1) /bin/cp 8.22/8.30/9.1 have almost same performance.
2) the case(the file is cached) is faster than the case(the file is not cached).

strace show that the logical of /bin/cp 8.30 and 9.1 are different. /bin/cp 8.30

    lseek(3, 198737920, SEEK_SET)           = 198737920
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    lseek(3, 198746112, SEEK_SET)           = 198746112
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

/bin/cp 9.1

    lseek(3, 880640, SEEK_DATA)             = 884736
    lseek(3, 884736, SEEK_HOLE)             = 888832
    lseek(3, 884736, SEEK_SET)              = 884736
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    lseek(3, 888832, SEEK_DATA)             = 892928
    lseek(3, 892928, SEEK_HOLE)             = 897024
    lseek(3, 892928, SEEK_SET)              = 892928
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

Do we need some job in btrfs to support /bin/cp 9.1 well, or /bin/cp 9.1 is just wrong?

Best Regards
Wang Yugui ([email protected])
2022/11/06

author
#1 lore raw

Hi,

a fix.

Quoted 7 lines

Hi,

newer /bin/cp have worse btrfs fiemap performance.

btrfs version: misc-next 750de989d367(the lastest of 2022/11/05) /bin/cp versions(run them on the same server)

    8.22 copy from centos/7.9

this

Quoted 1 line
    8.30 rocklinux/9.0

should be

	8.30 rocklinux/8

Best Regards
Wang Yugui ([email protected])
2022/11/06

Quoted 58 lines
    9.1 local build, https://kojipkgs.fedoraproject.org/packages/coreutils/9.1/8.eln121/

test case:
/bin/cp /mnt/test/file1 /dev/null
/mnt/test/file1 is created by https://lore.kernel.org/linux-btrfs/[email protected]/T/#T
and /mnt/test/file1 is 256M.

file is not cached: 'echo 3 >/proc/sys/vm/drop_caches' file is cached: run '/bin/cp /mnt/test/file1 /dev/null' again.

performance result(/bin/cp /mnt/test/file1 /dev/null):
/bin/cp 9.1

    file is not cached: 94.85(1:34.85)
    file is cached: 1982.43(33:02.43)
/bin/cp 8.30
    file is not cached: 1.48(0:01.48)
    file is cached:14.07(0:14.07)
/bin/cp 8.22
    file is not cached: 0.53(0:00.53)
    file is cached: 0.10(0:00.10)

as a compare, we test it on xfs too.
1) /bin/cp 8.22/8.30/9.1 have almost same performance.
2) the case(the file is cached) is faster than the case(the file is not cached).

strace show that the logical of /bin/cp 8.30 and 9.1 are different. /bin/cp 8.30

    lseek(3, 198737920, SEEK_SET)           = 198737920
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    lseek(3, 198746112, SEEK_SET)           = 198746112
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

/bin/cp 9.1

    lseek(3, 880640, SEEK_DATA)             = 884736
    lseek(3, 884736, SEEK_HOLE)             = 888832
    lseek(3, 884736, SEEK_SET)              = 884736
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    lseek(3, 888832, SEEK_DATA)             = 892928
    lseek(3, 892928, SEEK_HOLE)             = 897024
    lseek(3, 892928, SEEK_SET)              = 892928
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

Do we need some job in btrfs to support /bin/cp 9.1 well, or /bin/cp 9.1 is just wrong?

Best Regards
Wang Yugui ([email protected])
2022/11/06

#2 lore raw
In reply to Wang Yugui
Quoted 4 lines · Wang Yugui

Hi,

newer /bin/cp have worse btrfs fiemap performance.

New versions of coreutils (9.0 and 9.1) have worse cp performance, but it has nothing to do with fiemap.

This was pointed out in another thread ([1]) sometime ago, that starting with coreutils 9.0, cp no longer uses fiemap. That happened in coreutils' commit:

https://github.com/coreutils/coreutils/commit/26eccf6c98696c50f4416ba2967edc8676870716

You can also confirm that by checking that strace no longer shows any lines with:

ioctl(3, FS_IOC_FIEMAP, (...)

Quoted 26 lines

btrfs version: misc-next 750de989d367(the lastest of 2022/11/05) /bin/cp versions(run them on the same server)

    8.22 copy from centos/7.9
    8.30 rocklinux/9.0
    9.1 local build, https://kojipkgs.fedoraproject.org/packages/coreutils/9.1/8.eln121/

test case:
/bin/cp /mnt/test/file1 /dev/null
/mnt/test/file1 is created by https://lore.kernel.org/linux-btrfs/[email protected]/T/#T
and /mnt/test/file1 is 256M.

file is not cached: 'echo 3 >/proc/sys/vm/drop_caches' file is cached: run '/bin/cp /mnt/test/file1 /dev/null' again.

performance result(/bin/cp /mnt/test/file1 /dev/null):
/bin/cp 9.1

    file is not cached: 94.85(1:34.85)
    file is cached: 1982.43(33:02.43)
/bin/cp 8.30
    file is not cached: 1.48(0:01.48)
    file is cached:14.07(0:14.07)
/bin/cp 8.22
    file is not cached: 0.53(0:00.53)
    file is cached: 0.10(0:00.10)
Quoted 6 lines

as a compare, we test it on xfs too.
1) /bin/cp 8.22/8.30/9.1 have almost same performance.
2) the case(the file is cached) is faster than the case(the file is not cached).

strace show that the logical of /bin/cp 8.30 and 9.1 are different.

Yes, as pointed out before.

Before coreutils 9.0, cp used fiemap to determine where holes and extents are in a file.
However starting with 9.0 it uses SEEK_DATA/SEEK_HOLE from lseek to do that, completely
dropping fiemap usage.

Quoted 24 lines

/bin/cp 8.30

    lseek(3, 198737920, SEEK_SET)           = 198737920
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    lseek(3, 198746112, SEEK_SET)           = 198746112
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

/bin/cp 9.1

    lseek(3, 880640, SEEK_DATA)             = 884736
    lseek(3, 884736, SEEK_HOLE)             = 888832
    lseek(3, 884736, SEEK_SET)              = 884736
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    lseek(3, 888832, SEEK_DATA)             = 892928
    lseek(3, 892928, SEEK_HOLE)             = 897024
    lseek(3, 892928, SEEK_SET)              = 892928
    write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(3, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    write(4, "a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
Quoted 2 lines

Do we need some job in btrfs to support /bin/cp 9.1 well, or /bin/cp 9.1 is just wrong?

There's nothing wrong with cp using lseek's SEEK_DATA/HOLE features. It's just that the data/hole seek calls are where cp is spending most time, so the way to go is
to improve the efficiency of those calls in btrfs. I'll do some work to make it better.

Keep in mind that the case being tested is pretty much the worst possible scenario. It's an extremely
sparse file, where each extent has a hole before it and another hole after it. More often than not,
there are significantly more extents than holes in a typical file.

Thanks.

[1] https://lore.kernel.org/linux-btrfs/CAL3q7H5NSVicm7nYBJ7x8fFkDpno8z3PYt5aPU43Bajc1H0h1Q@mail.gmail.com/

Quoted 6 lines

Best Regards
Wang Yugui ([email protected])
2022/11/06