9.3.4: resync between two Secondaries stuck at 100% - flush request never acked (2 volumes, new-peer before attach)

Hello,

On a two-volume resource, a resync between two Secondaries reaches 100% (out-of-sync 0) and never completes: the target stays Inconsistent/SyncTarget on both volumes indefinitely. It happens when the source was brought up with a plain "drbdadm up" and its second volume's peer bitmap_dagtag is higher than its first volume's. We hit it in production (two standby hosts, both Secondary while the Primary OS is down) and have reproduced it in two VMs. Details, debugfs output and our reading of the source are below. Please correct the analysis if we have it wrong.

Environment
-----------

- drbd-dkms 9.3.4-1ppa1~resolute1 (GIT-hash e59b287b199f7224468af85b68a10e83c68213f5), drbd-utils 9.35.0-1ppa1~resolute1, LINBIT PPA
- Ubuntu 26.04, kernel 7.0.0-38-generic
- 2 nodes, protocol C, 1 resource with 2 volumes, external metadata, auto-promote no

    resource r0 {
        options { auto-promote no; }
        net { protocol C; }
        on pm1 {
            node-id 0;
            address 10.10.10.1:7799;
            volume 0 { device /dev/drbd0; disk /dev/disk/by-partlabel/drbd-r0; meta-disk /dev/disk/by-partlabel/drbd-r0-md; }
            volume 1 { device /dev/drbd1; disk /dev/disk/by-partlabel/drbd-r1; meta-disk /dev/disk/by-partlabel/drbd-r1-md; }
        }
        on pm2 {
            node-id 1;
            address 10.10.10.2:7799;
            volume 0 { device /dev/drbd0; disk /dev/disk/by-partlabel/drbd-r0; meta-disk /dev/disk/by-partlabel/drbd-r0-md; }
            volume 1 { device /dev/drbd1; disk /dev/disk/by-partlabel/drbd-r1; meta-disk /dev/disk/by-partlabel/drbd-r1-md; }
        }
    }

(Production also has al-extents 6433 and resync-controller settings; they are not needed to reproduce.)

Reproduction
------------

Start with both nodes connected and UpToDate.

pm1: make the resource dagtag non-zero
pm1# drbdadm primary r0
pm1# dd if=/dev/urandom of=/dev/drbd0 bs=1M count=10 oflag=direct pm1# dd if=/dev/urandom of=/dev/drbd1 bs=1M count=10 oflag=direct

pm2 goes away
pm2# drbdadm down r0

pm1 writes vol0 FIRST, then vol1, so vol1's peer bitmap_dagtag ends up higher
pm1# dd if=/dev/urandom of=/dev/drbd0 bs=1M count=20 oflag=direct pm1# dd if=/dev/urandom of=/dev/drbd1 bs=1M count=20 oflag=direct pm1# drbdadm secondary r0 && drbdadm down r0

both come back as Secondary with a plain "up" (new-peer before attach)
pm1# drbdadm up r0
pm2# drbdadm up r0

Peer slot 1 in pm1's metadata before the "up" (after apply-al):

    vol0 peer[1]  bitmap-dagtag 0x000000000002D000
    vol1 peer[1]  bitmap-dagtag 0x0000000000037000

Result: pm2 is SyncTarget on both volumes with out-of-sync 0 and stays there (we watched for several minutes; in production it stayed for about 15 minutes until we intervened).

debugfs, /sys/kernel/debug/drbd/resources/r0/connections/<peer>/debug:

pm1 (SyncSource), connection pm2:
resource->dagtag_sector: 225280 (0x37000, from vol1's metadata)
last_peer_ack_dagtag_seen: 0
flush_requests_dagtag: 184320 (0x2D000 = vol0's dagtag = dagtag_before_attach)
flush_sequence: 1
flush_forward_sent_mask: (empty: never acked)

pm2 (SyncTarget), connection pm1:
pending_flush_mask: pm1,

"drbdadm disconnect r0; drbdadm connect r0" on pm2 starts a 0 KB resync that also never finishes. pm1 afterwards still shows
last_peer_ack_dagtag_seen: 0 and flush_requests_dagtag: 184320, now with flush_sequence: 2.

"drbdadm primary r0" on pm1 plus a single 4 KiB write moves last_peer_ack_dagtag_seen to 225288, and the resync finishes at once.

Controls (same VMs):

- Vol0 written last (so vol0's peer bitmap_dagtag is higher), plain "up": the resync finishes. flush_requests_dagtag = 0, while last_peer_ack_dagtag_seen is also 0.
- Equal or zero bitmap dagtags (the peer left before any writes):
finishes.
- Original order, but pm1 brought up with "drbdadm adjust --skip-net r0" and then "drbdadm adjust r0" (attach before new-peer): finishes in 1 s. last_peer_ack_dagtag_seen = 307208 >= flush_requests_dagtag = 266248.

Analysis (from reading the 9.3.4 source)
----------------------------------------

1. "drbdadm up" issues new-resource, new-minor, new-peer, new-path, peer-device-options, attach, connect (per "drbdadm -d up"), so new-peer runs before any attach.

2. new-peer (drbd_nl.c ~5118) initialises connection->last_peer_ack_dagtag_seen = resource->dagtag_sector, "so that flushes can be acked even if no further writes occur". At that point the dagtag is still 0.

3. Each attach calls update_resource_dagtag() (drbd_nl.c ~2843). The first volume raises dagtag_sector from 0 to its max peer bitmap_dagtag (dagtag_before_attach = 0). If the second volume's bitmap_dagtag is higher, it sets dagtag_before_attach to the first volume's (non-zero) value and dagtag_from_backing_dev to its own.

4. When the SyncTarget sends P_FLUSH_REQUESTS, receive_flush_requests() (drbd_receiver.c ~5176) sees dagtag_sector == dagtag_from_backing_dev, so it uses dagtag_before_attach (the non-zero vol0 value) as flush_requests_dagtag.

5. drbd_check_flush_dagtag_reached() (drbd_receiver.c ~12408) acks only when last_peer_ack_dagtag_seen >= flush_requests_dagtag. With no Primary anywhere, last_peer_ack_dagtag_seen never moves from 0, so the ack never comes.

6. On the target, drbd_resync_check_finished() (drbd_sender.c ~1078) waits on drbd_any_flush_pending(), which is resource-wide, so both volumes stay SyncTarget. (In production, one volume had already enabled peer replication and the other had not.)

With one volume, dagtag_before_attach stays 0 and the ack always passes, which is why we only started seeing this after moving from one volume to two.

Possible fixes (your call): when an attach raises dagtag_sector, also bring last_peer_ack_dagtag_seen up on existing connections that have no Primary peer; or clamp the flush target in receive_flush_requests() so it cannot exceed what this node could ever see acked when it has no Primary peers.

Workaround
----------

On the standby nodes we attach before creating the peer: an ExecStartPre= for [email protected] runs "drbdadm adjust --skip-net r0" ahead of its normal "adjust".

Happy to test patches or gather more data; the VM reproduction takes about two minutes.

Thanks,
Dominic Ferard

Hi Dominic,

thank you for the excellent report. Your analysis is correct. We found the same bug independently, and the fix is already in git:

drbd-9.2: 24374f123a95 ("drbd: keep the dagtag of the last request
across several attaches")
https://github.com/LINBIT/drbd/commit/24374f123a95877956c3c95a5080b6afae75b28c

master: eec2d913473b (merge of the above into drbd-9.3)
https://github.com/LINBIT/drbd/commit/eec2d913473bf00370b1049e549f10ac0e6181cd

Only 9.3.4 is affected in practice. It is the first release that also sends the flush to Secondaries. Earlier releases flush only Primary peers, and a Primary's own writes move the peer acks forward.

The fix will be in 9.3.5, which we plan to release on November 9th.

If you want to confirm the fix in your VMs before then, the commit above applies on top of 9.3.4.

Best regards,
Philipp

Am Sat, Oct 24, 2026 at 05:11:08PM +0100 schrieb Dominic Ferard:

[...]