Skip to contentSkip to conversation

DRBD 8.4.3 : create-md possibly broken for fixed sized (shared) meta devices

Drbd-dev Tijs Van Buggenhout 2 participants 3 messages Mar 6, 2013 – Mar 7, 2013 DRBD driver

lore
Share & tools
lore

Tijs Van Buggenhout

author
#0 lore raw

Hi,

I recently installed drbd 8.4.3 packages and ran into problems when trying to create a drbd resource using fixed size drbd meta device ided by an index. Investigating further, it seems that when using a flexible meta disk, there is no problem whatsoever.
I also used the same configuration file/set of commands on the previous version (8.4.2) for fixed size meta-disk, which worked without problems.

The following excerpt from the console log shows how I first create a drbd resource (drbd0) with flexible meta disk (/dev/lv/drbd-meta-84). Afterwards I undo the configuration, and try to do the same for fixed drbd meta disk, which fails (every time). Then I roll back to the previous version of drbd utils/module, and try the setup with fixed drbd meta disk again, which succeeds.

(master.domain.com):~# lvdisplay lv
  --- Logical volume ---
  LV Path                /dev/lv/data
  LV Name                data
  VG Name                lv
  LV UUID                A1WAT3-uWDN-6ch5-BCnc-KTyJ-7ZiT-FdKhDZ
  LV Write Access        read/write
  LV Creation host, time master.domain.com, 2013-03-07 01:18:00 +0100
  LV Status              available
  # open                 0
  LV Size                512.00 MiB
  Current LE             128
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

--- Logical volume ---

  LV Path                /dev/lv/drbd-meta-84
  LV Name                drbd-meta-84
  VG Name                lv
  LV UUID                6fW32r-94t8-G8vZ-I0ME-ElJi-LeWC-u4Hdik
  LV Write Access        read/write
  LV Creation host, time master.domain.com, 2013-03-07 01:18:35 +0100
  LV Status              available
  # open                 0
  LV Size                128.00 MiB
  Current LE             32
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
(master.domain.com):~# cat /etc/drbd.conf 
#
# drbd.conf (generated, do not modify)
#
global {
        # minor-count 32;
        # dialog-refresh 5; # 5 seconds
        # disable-ip-verification;

        # Participate in DRBD's online usage counter at http://usage.drbd.org
        usage-count no;
}
common {
        disk { resync-rate 10M; }
}
resource drbd0 {
        protocol C;

        handlers {
                # pri-on-incon-degr "echo o > /proc/sysrq-trigger; halt -f";
                # pri-lost-after-sb "echo o > /proc/sysrq-trigger; halt -f";
                # local-io-error "echo o > /proc/sysrq-trigger; halt -f";
        }

        startup {
                # wfc-timeout  0;
                # degr-wfc-timeout 60;
                # outdated-wfc-timeout 0;
                # wait-after-sb no;
                # stacked-timeouts no;
                # become-primary-on master.domain.com;
        }

        disk {
                fencing dont-care;
                on-io-error detach;
                al-extents 1237;
                al-updates yes;
                resync-rate 700K;
        }

        net {
                after-sb-0pri discard-younger-primary;
                after-sb-1pri consensus;
                after-sb-2pri disconnect;
                rr-conflict disconnect;
        }

        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.179:7788;
                flexible-meta-disk /dev/lv/drbd-meta-84;
        }

        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.180:7788;
                flexible-meta-disk /dev/lv/drbd-meta-84;
        }
}
(master.domain.com):~# dd if=/dev/zero of=/dev/lv/drbd-meta-84 bs=$[1024*1024] 
count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 10.7637 s, 12.5 MB/s
(master.domain.com):~# modprobe drbd
(master.domain.com):~# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build, 
2013-03-05 15:10:01
(master.domain.com):~# drbdsetup new-resource drbd0
(master.domain.com):~# drbdsetup new-minor drbd0 /dev/drbd0 0
(master.domain.com):~# drbdadm create-md drbd0 -vvvvvvvvvv
drbdmeta 0 v08 /dev/lv/drbd-meta-84 flex-external create-md 
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
drbdmeta 0 v08 /dev/lv/drbd-meta-84 flex-external write-dev-uuid 
000000000000000D
(master.domain.com):~# drbdmeta /dev/drbd0 v08 /dev/lv/drbd-meta-84 0 dump-md 
-vvvvvvvvvv
hard_sect_size is 512 Byte
 v08_md_disk_to_cpu        : pread(4, ...,  4096,           0)
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 04  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 0d  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 80 83 74 02 6b  |.............t.k|
00000040  00 04 00 00 00 00 00 08  00 00 01 01 00 00 00 48  |...............H|
00000050  00 00 10 00 00 00 00 00  00 00 00 01 00 00 00 08  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00001000
# DRBD meta data dump
# 2013-03-07 07:17:01 +0100 [1362637021]
# master.domain.com> drbdmeta -vvvvvvvvvv /dev/drbd0 v08 /dev/lv/drbd-meta-84 
0 dump-md
#

version "v08";

# md_size_sect 262144
# md_offset 0
# al_offset 4096
# bm_offset 36864
uuid {
    0x0000000000000004; 0x0000000000000000; 0x0000000000000000; 
0x0000000000000000;
    flags 0x00000080;
}
# al-extents 257;
la-size-sect 0;
bm-byte-per-bit 4096;
device-uuid 0x000000000000000D;
la-peer-max-bio-size 0;
# bm-bytes 0;
bm {
}
# bits-set 0;
# al {
 printf_al                 : pread(4, ..., 32768,        4096)
#     block  0 { INITIALIZED }
#     block  1 { INITIALIZED }
#     block  2 { INITIALIZED }
#     block  3 { INITIALIZED }
#     block  4 { INITIALIZED }
#     block  5 { INITIALIZED }
#     block  6 { INITIALIZED }
#     block  7 { INITIALIZED }
# }
(master.domain.com):~# drbdadm attach drbd0
(master.domain.com):~# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build, 
2013-03-05 15:10:01
 0: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:524160
(master.domain.com):~# drbdadm detach drbd0
(master.domain.com):~# drbdadm down drbd0
(master.domain.com):~# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build, 
2013-03-05 15:10:01
(master.domain.com):~# rmmod drbd
(master.domain.com):~# cat /etc/drbd.conf 
#
# drbd.conf (generated, do not modify)
#
global {
        # minor-count 32;
        # dialog-refresh 5; # 5 seconds
        # disable-ip-verification;

        # Participate in DRBD's online usage counter at http://usage.drbd.org
        usage-count no;
}
common {
        disk { resync-rate 10M; }
}
resource drbd0 {
        protocol C;

        handlers {
                # pri-on-incon-degr "echo o > /proc/sysrq-trigger; halt -f";
                # pri-lost-after-sb "echo o > /proc/sysrq-trigger; halt -f";
                # local-io-error "echo o > /proc/sysrq-trigger; halt -f";
        }

        startup {
                # wfc-timeout  0;
                # degr-wfc-timeout 60;
                # outdated-wfc-timeout 0;
                # wait-after-sb no;
                # stacked-timeouts no;
                # become-primary-on master.domain.com;
        }

        disk {
                fencing dont-care;
                on-io-error detach;
                al-extents 1237;
                al-updates yes;
                resync-rate 700K;
        }

        net {
                after-sb-0pri discard-younger-primary;
                after-sb-1pri consensus;
                after-sb-2pri disconnect;
                rr-conflict disconnect;
        }

        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.179:7788;
                meta-disk /dev/lv/drbd-meta-84 [0];
        }

        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.180:7788;
                meta-disk /dev/lv/drbd-meta-84 [0];
        }
}
(master.domain.com):~# dd if=/dev/zero of=/dev/lv/drbd-meta-84 bs=$[1024*1024] 
count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 10.6723 s, 12.6 MB/s
(master.domain.com):~# modprobe drbd
(master.domain.com):~# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build, 
2013-03-05 15:10:01
(master.domain.com):~# drbdsetup new-resource drbd0
(master.domain.com):~# drbdsetup new-minor drbd0 /dev/drbd0 0
(master.domain.com):~# drbdadm create-md drbd0 -vvvvvvvvvv
drbdmeta 0 v08 /dev/lv/drbd-meta-84 0 create-md 
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
drbdmeta 0 v08 /dev/lv/drbd-meta-84 0 write-dev-uuid 000000000000000D 
(master.domain.com):~# drbdmeta /dev/drbd0 v08 /dev/lv/drbd-meta-84 0 dump-md 
-vvvvvvvvvv
hard_sect_size is 512 Byte
 v08_md_disk_to_cpu        : pread(4, ...,  4096,           0)
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 04  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 0d  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 80 83 74 02 6b  |.............t.k|
00000040  00 04 00 00 00 00 00 08  00 00 01 01 00 00 00 48  |...............H|
00000050  00 00 10 00 00 00 00 00  00 00 00 01 00 00 00 08  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00001000
# DRBD meta data dump
# 2013-03-07 07:22:35 +0100 [1362637355]
# master.domain.com> drbdmeta -vvvvvvvvvv /dev/drbd0 v08 /dev/lv/drbd-meta-84 
0 dump-md
#

version "v08";

# md_size_sect 262144
# md_offset 0
# al_offset 4096
# bm_offset 36864
uuid {
    0x0000000000000004; 0x0000000000000000; 0x0000000000000000; 
0x0000000000000000;
    flags 0x00000080;
}
# al-extents 257;
la-size-sect 0;
bm-byte-per-bit 4096;
device-uuid 0x000000000000000D;
la-peer-max-bio-size 0;
# bm-bytes 0;
bm {
}
# bits-set 0;
# al {
 printf_al                 : pread(4, ..., 32768,        4096)
#     block  0 { INITIALIZED }
#     block  1 { INITIALIZED }
#     block  2 { INITIALIZED }
#     block  3 { INITIALIZED }
#     block  4 { INITIALIZED }
#     block  5 { INITIALIZED }
#     block  6 { INITIALIZED }
#     block  7 { INITIALIZED }
# }
(master.domain.com):~# drbdadm attach drbd0
0: Failure: (119) No valid meta-data signature found.
        ==> Use 'drbdadm create-md res' to initialize meta-data area. <==

Command 'drbdsetup attach 0 /dev/lv/data /dev/lv/drbd-meta-84 0 --resync- rate=700K --al-updates=yes --al-extents=1237 --on-io-error=detach -- fencing=dont-care' terminated with exit code 10

(master.domain.com):~# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build, 
2013-03-05 15:10:01
 0: cs:Unconfigured
(master.domain.com):~# drbdadm down drbd0
(master.domain.com):~# rmmod drbd
.... reinstall drbd 8.4.2 modules/tools ...
(master.domain.com):~# dd if=/dev/zero of=/dev/lv/drbd-meta-84 bs=$[1024*1024] 
count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 10.7561 s, 12.5 MB/s
(master.domain.com):~# depmod -a
(master.domain.com):~# modprobe drbd
(master.domain.com):~# cat /proc/drbd 
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@build, 
2013-03-06 10:15:58
(master.domain.com):~# cat /etc/drbd.conf 
#
# drbd.conf (generated, do not modify)
#
global {
        # minor-count 32;
        # dialog-refresh 5; # 5 seconds
        # disable-ip-verification;

        # Participate in DRBD's online usage counter at http://usage.drbd.org
        usage-count no;
}
common {
        disk { resync-rate 10M; }
}
resource drbd0 {
        protocol C;

        handlers {
                # pri-on-incon-degr "echo o > /proc/sysrq-trigger; halt -f";
                # pri-lost-after-sb "echo o > /proc/sysrq-trigger; halt -f";
                # local-io-error "echo o > /proc/sysrq-trigger; halt -f";
        }

        startup {
                # wfc-timeout  0;
                # degr-wfc-timeout 60;
                # outdated-wfc-timeout 0;
                # wait-after-sb no;
                # stacked-timeouts no;
                # become-primary-on master.domain.com;
        }

        disk {
                fencing dont-care;
                on-io-error detach;
                al-extents 1237;
                #al-updates yes;
                resync-rate 700K;
        }

        net {
                after-sb-0pri discard-younger-primary;
                after-sb-1pri consensus;
                after-sb-2pri disconnect;
                rr-conflict disconnect;
        }

        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.179:7788;
                meta-disk /dev/lv/drbd-meta-84 [0];
        }

        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.180:7788;
                meta-disk /dev/lv/drbd-meta-84 [0];
        }
}
(master.domain.com):~# drbdsetup new-resource drbd0
(master.domain.com):~# drbdsetup new-minor drbd0 /dev/drbd0 0
(master.domain.com):~# drbdadm create-md drbd0 -vvvvvvvvvv
drbdmeta 0 v08 /dev/lv/drbd-meta-84 0 create-md 
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
drbdmeta 0 v08 /dev/lv/drbd-meta-84 0 write-dev-uuid 000000000000000D 
(master.domain.com):~# drbdmeta /dev/drbd0 v08 /dev/lv/drbd-meta-84 0 dump-md 
-vvvvvvvvvv
hard_sect_size is 512 Byte
 v08_md_disk_to_cpu        : pread(4, ...,  4096,           0)
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 04  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 0d  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 83 74 02 6b  |.............t.k|
00000040  00 04 00 00 00 00 00 08  00 00 01 01 00 00 00 48  |...............H|
00000050  00 00 10 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00001000
# DRBD meta data dump
# 2013-03-07 07:32:58 +0100 [1362637978]
# master.domain.com> drbdmeta -vvvvvvvvvv /dev/drbd0 v08 /dev/lv/drbd-meta-84 
0 dump-md
#

version "v08";

# md_size_sect 262144
# md_offset 0
# al_offset 4096
# bm_offset 36864
uuid {
    0x0000000000000004; 0x0000000000000000; 0x0000000000000000; 
0x0000000000000000;
    flags 0x00000000;
}
# al-extents 257;
la-size-sect 0;
bm-byte-per-bit 4096;
device-uuid 0x000000000000000D;
la-peer-max-bio-size 0;
# bm-bytes 0;
bm {
}
# bits-set 0;
# al {
 printf_al                 : pread(4, ..., 32768,        4096)
#     block  0 { INITIALIZED }
#     block  1 { INITIALIZED }
#     block  2 { INITIALIZED }
#     block  3 { INITIALIZED }
#     block  4 { INITIALIZED }
#     block  5 { INITIALIZED }
#     block  6 { INITIALIZED }
#     block  7 { INITIALIZED }
# }
(master.domain.com):~# drbdadm attach drbd0
(master.domain.com):~# cat /proc/drbd 
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@build, 
2013-03-06 10:15:58
 0: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:524160
(master.domain.com):~# drbdadm detach drbd0
(master.domain.com):~# drbdadm down drbd0
(master.domain.com):~# rmmod drbd

Am I doing something wrong, or did I misconfigure drbd, such that the setup of a drbd resource with fixed size meta disk fails in version 8.4.3, or can this be some kind of regression problem/bug?

Best regards,
Tijs

maintainer
#1 lore raw
Quoted 35 lines · Tijs Van Buggenhout

Hi,

I recently installed drbd 8.4.3 packages and ran into problems when trying to create a drbd resource using fixed size drbd meta device ided by an index. Investigating further, it seems that when using a flexible meta disk, there is no problem whatsoever.
I also used the same configuration file/set of commands on the previous version (8.4.2) for fixed size meta-disk, which worked without problems.

The following excerpt from the console log shows how I first create a drbd resource (drbd0) with flexible meta disk (/dev/lv/drbd-meta-84). Afterwards I undo the configuration, and try to do the same for fixed drbd meta disk, which fails (every time). Then I roll back to the previous version of drbd utils/module, and try the setup with fixed drbd meta disk again, which succeeds.

resource drbd0 {
        on master.domain.com {
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.179:7788;
                meta-disk /dev/lv/drbd-meta-84 [0];
(master.domain.com):~# dd if=/dev/zero of=/dev/lv/drbd-meta-84 bs=$[1024*1024] 
(master.domain.com):~# modprobe drbd
(master.domain.com):~# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build, 
2013-03-05 15:10:01
(master.domain.com):~# drbdsetup new-resource drbd0
(master.domain.com):~# drbdsetup new-minor drbd0 /dev/drbd0 0
(master.domain.com):~# drbdadm create-md drbd0 -vvvvvvvvvv
(master.domain.com):~# drbdadm attach drbd0
0: Failure: (119) No valid meta-data signature found.
Quoted 1 line
        ==> Use 'drbdadm create-md res' to initialize meta-data area. <==

D'oh.
That is embarrassing.

I wonder why the regression tests using "old style indexed fixed size external meta data" have not been exercised.

http://git.drbd.org/gitweb.cgi?p=drbd-8.4.git;a=commitdiff;h=82cf5feada2e4d0f536af5f6f377bd0f3fac1236

Thanks,

Signature · 5 lines
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
author
#2 lore raw
Quoted 43 lines · On Thursday 07 March 2013 10:31:37 Lars Ellenberg wrote:
Quoted 37 lines · Tijs Van Buggenhout

Hi,

I recently installed drbd 8.4.3 packages and ran into problems when trying to create a drbd resource using fixed size drbd meta device ided by an index. Investigating further, it seems that when using a flexible meta disk, there is no problem whatsoever.
I also used the same configuration file/set of commands on the previous version (8.4.2) for fixed size meta-disk, which worked without problems.

The following excerpt from the console log shows how I first create a drbd resource (drbd0) with flexible meta disk (/dev/lv/drbd-meta-84). Afterwards I undo the configuration, and try to do the same for fixed drbd meta disk, which fails (every time). Then I roll back to the previous version of drbd utils/module, and try the setup with fixed drbd meta disk again, which succeeds.

resource drbd0 {

        on master.domain.com {
        
                device /dev/drbd0;
                disk /dev/lv/data;
                address 172.16.1.179:7788;
                meta-disk /dev/lv/drbd-meta-84 [0];
(master.domain.com):~# dd if=/dev/zero of=/dev/lv/drbd-meta-84
bs=$[1024*1024] (master.domain.com):~# modprobe drbd
(master.domain.com):~# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@build,
2013-03-05 15:10:01
(master.domain.com):~# drbdsetup new-resource drbd0
(master.domain.com):~# drbdsetup new-minor drbd0 /dev/drbd0 0
(master.domain.com):~# drbdadm create-md drbd0 -vvvvvvvvvv
(master.domain.com):~# drbdadm attach drbd0
0: Failure: (119) No valid meta-data signature found.
Quoted 1 line
        ==> Use 'drbdadm create-md res' to initialize meta-data area. <==

D'oh.
That is embarrassing.

Quoted 7 lines

I wonder why the regression tests using "old style indexed fixed size external meta data" have not been exercised.

http://git.drbd.org/gitweb.cgi?p=drbd-8.4.git;a=commitdiff;h=82cf5feada2e4d0
f536af5f6f377bd0f3fac1236

Thanks,

Patch is working, thanks for the quick response/fix!

Is a new release of drbd 8.4 series to be expected in the near future?

Regards,
Tijs