btrfs send receive: No space left on device

Hello,
i have new 32GB SSD in my intel nuc, installed debian9 on it, using btrfs as a rootfs. Then i created subvolumes /system and /home and moved system there.

System was installed using kernel 4.9.x and filesystem created using btrfs-progs 4.7.x Details follow:
main filesystem

# btrfs filesystem usage /mnt/btrfs/ssd/
Overall:
    Device size:                  29.08GiB
    Device allocated:              4.28GiB
    Device unallocated:           24.80GiB
    Device missing:                  0.00B
    Used:                          2.54GiB
    Free (estimated):             26.32GiB      (min: 26.32GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:4.00GiB, Used:2.48GiB
/dev/sda3 4.00GiB

Metadata,single: Size:256.00MiB, Used:61.05MiB
/dev/sda3 256.00MiB

System,single: Size:32.00MiB, Used:16.00KiB
/dev/sda3 32.00MiB

Unallocated:
/dev/sda3 24.80GiB

#/etc/fstab

UUID=d801da52-813d-49da-bdda-87fc6363e0ac       /mnt/btrfs/ssd          btrfs noatime,space_cache=v2,compress=lzo,commit=300,subvolid=5         0       0
UUID=d801da52-813d-49da-bdda-87fc6363e0ac       /                       btrfs   noatime,space_cache=v2,compress=lzo,commit=300,subvol=/system         0       0
UUID=d801da52-813d-49da-bdda-87fc6363e0ac       /home                   btrfs   noatime,space_cache=v2,compress=lzo,commit=300,subvol=/home         0       0

-----
Then i installed kernel from backports:
4.18.0-0.bpo.1-amd64 #1 SMP Debian 4.18.6-1~bpo9+1 and btrfs-progs 4.17

For backups , i have created 16GB iscsi device on my qnap and mounted it, created filesystem, mounted like this:
LABEL=backup /mnt/btrfs/backup btrfs noatime,space_cache=v2,compress=lzo,subvolid=5,nofail,noauto 0 0

After send-receive operation on /home subvolume, usage looks like this:

# btrfs filesystem usage /mnt/btrfs/backup/
Overall:
    Device size:                  16.00GiB
    Device allocated:              1.27GiB
    Device unallocated:           14.73GiB
    Device missing:                  0.00B
    Used:                        844.18MiB
    Free (estimated):             14.92GiB      (min: 14.92GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:1.01GiB, Used:833.36MiB
/dev/sdb 1.01GiB

Metadata,single: Size:264.00MiB, Used:10.80MiB
/dev/sdb 264.00MiB

System,single: Size:4.00MiB, Used:16.00KiB
/dev/sdb 4.00MiB

Unallocated:
/dev/sdb 14.73GiB

Problem is, during send-receive of system subvolume, it runs out of space:

# btrbk run /mnt/btrfs/ssd/system/ -v      
btrbk command line client, version 0.26.1  (Wed Oct 17 09:51:20 2018)
Using configuration: /etc/btrbk/btrbk.conf
Using transaction log: /var/log/btrbk.log
Creating subvolume snapshot for: /mnt/btrfs/ssd/system
[snapshot] source: /mnt/btrfs/ssd/system
[snapshot] target: /mnt/btrfs/ssd/_snapshots/system.20181017T0951
Checking for missing backups of subvolume "/mnt/btrfs/ssd/system" in "/mnt/btrfs/backup/"
Creating subvolume backup (send-receive) for: /mnt/btrfs/ssd/_snapshots/system.20181016T2034
No common parent subvolume present, creating full backup...
[send/receive] source: /mnt/btrfs/ssd/_snapshots/system.20181016T2034
[send/receive] target: /mnt/btrfs/backup/system.20181016T2034
mbuffer: error: outputThread: error writing to <stdout> at offset 0x4b5bd000: Broken pipe
mbuffer: warning: error during output to <stdout>: Broken pipe
WARNING: [send/receive] (send=/mnt/btrfs/ssd/_snapshots/system.20181016T2034, receive=/mnt/btrfs/backup) At subvol /mnt/btrfs/ssd/_snapshots/system.20181016T2034
WARNING: [send/receive] (send=/mnt/btrfs/ssd/_snapshots/system.20181016T2034, receive=/mnt/btrfs/backup) At subvol system.20181016T2034
ERROR: rename o77417-5519-0 -> lib/modules/4.18.0-0.bpo.1-amd64/kernel/drivers/watchdog/pcwd_pci.ko failed: No space left on device
ERROR: Failed to send/receive btrfs subvolume: /mnt/btrfs/ssd/_snapshots/system.20181016T2034  -> /mnt/btrfs/backup
[delete] options: commit-after
[delete] target: /mnt/btrfs/backup/system.20181016T2034
WARNING: Deleted partially received (garbled) subvolume: /mnt/btrfs/backup/system.20181016T2034
ERROR: Error while resuming backups, aborting
Created 0/2 missing backups
WARNING: Skipping cleanup of snapshots for subvolume "/mnt/btrfs/ssd/system", as at least one target aborted earlier
Completed within: 116s  (Wed Oct 17 09:53:16 2018)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.26.1)

    Date:   Wed Oct 17 09:51:20 2018
    Config: /etc/btrbk/btrbk.conf
    Filter: subvolume=/mnt/btrfs/ssd/system

Legend:

    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)

--------------------------------------------------------------------------------
/mnt/btrfs/ssd/system
+++ /mnt/btrfs/ssd/_snapshots/system.20181017T0951
!!! /mnt/btrfs/backup/system.20181016T2034
!!! Target "/mnt/btrfs/backup" aborted: Failed to send/receive subvolume

NOTE: Some errors occurred, which may result in missing backups! Please check warning and error messages above.
-----------------------------

With compression=none or compression=zlib , send-receive runs ok

Usage looks like this (after transfer):

# btrfs filesystem usage /mnt/btrfs/backup/
Overall:
    Device size:                  16.00GiB
    Device allocated:              2.27GiB
    Device unallocated:           13.73GiB
    Device missing:                  0.00B
    Used:                          1.64GiB
    Free (estimated):             14.15GiB      (min: 14.15GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:2.01GiB, Used:1.59GiB
/dev/sdb 2.01GiB

Metadata,single: Size:264.00MiB, Used:54.20MiB
/dev/sdb 264.00MiB

System,single: Size:4.00MiB, Used:16.00KiB
/dev/sdb 4.00MiB

Unallocated:
/dev/sdb 13.73GiB

I will stick to zlib for now, i just wanted to report this

Thanks,
Libor

I have seen similar behavior when doing a btrfs receive onto an compression=lzo or compression=zstd mounted filesystem on an SD-card. (I was cloning the rootfilesystem from an older SBC to a newly bought one.) With multiple tries, I noticed that the receive process failed at about the same point (a folder with .h-files), but not always the same file. I think that some pipeline or queue overflow/failure at the lower levels (maybe even below btrfs) causes this, but I have not futher looked at it.

The workaround was to have the receiving filesystem first as a loop-mounted sparse image on the laptop that I used for the cloning. And then writing the image to the designated partition on the SD-card with dd_rescue.

The laptop ran Opensuse Tumbleweed (about 2 weeks ago) with a snapshot not older than 1 month. If needed, I can figure out in more detail what kernel etc, but it was kernel 4.18.x and progs 4.17.x series.