RAID0 wrong (raw) device?
I have 2 identical servers with 2 x 2 Hitachi (HGST) SATA disks (and some other disks) which are mirrored with drbd.
On top of this drbd setup I have created a btrfs RAID0 filesystem. The problem now is, that btrfs shows the raw device instead of the drbd device.
root@toy02:~# mkfs.btrfs /dev/drbd2 /dev/drbd3 root@toy02:~# mount btrfs filesystem label /dev/drbd2 data root@toy02:~# mount /dev/drbd2 /data
root@toy02:~# df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdb btrfs 3906909856 140031696 3765056176 4% /data
root@toy02:~# btrfs filesystem show /data
Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b
Total devices 2 FS bytes used 129.81GiB
devid 3 size 1.82TiB used 67.03GiB path /dev/drbd2
devid 4 size 1.82TiB used 67.03GiB path /dev/sdbBtrfs v3.12
==> btrfs shows the wrong (raw) device /dev/sdb instead of /dev/drbd3 !
root@toy02:~# uname -a; lsb_release -a Linux toy02 3.13.0-61-generic #100-Ubuntu SMP Wed Jul 29 11:21:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty
root@toy02:~# find /dev -ls | grep drbd 47453 0 brw-rw---- 1 root disk Aug 12 14:51 /dev/drbd3 47433 0 brw-rw---- 1 root disk Aug 11 14:00 /dev/drbd2 14706 0 drwxr-xr-x 4 root root 80 Aug 10 14:17 /dev/drbd 14713 0 drwxr-xr-x 2 root root 100 Aug 12 13:40 /dev/drbd/by-res 41685 0 lrwxrwxrwx 1 root root 11 Aug 12 14:51 /dev/drbd/by-res/d3 -> ../../drbd3 42759 0 lrwxrwxrwx 1 root root 11 Aug 11 14:00 /dev/drbd/by-res/d2 -> ../../drbd2 14707 0 drwxr-xr-x 3 root root 60 Aug 10 14:17 /dev/drbd/by-disk 14708 0 drwxr-xr-x 3 root root 60 Aug 10 14:17 /dev/drbd/by-disk/disk 14709 0 drwxr-xr-x 2 root root 100 Aug 12 13:40 /dev/drbd/by-disk/disk/by-id 41682 0 lrwxrwxrwx 1 root root 17 Aug 12 14:51 /dev/drbd/by-disk/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../../../drbd3 42756 0 lrwxrwxrwx 1 root root 17 Aug 11 14:00 /dev/drbd/by-disk/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2XX -> ../../../../drbd2 41681 0 lrwxrwxrwx 1 root root 8 Aug 12 14:51 /dev/block/147:3 -> ../drbd3 42755 0 lrwxrwxrwx 1 root root 8 Aug 11 14:00 /dev/block/147:2 -> ../drbd2
root@toy02:~# find /dev -ls | grep HGST 41682 0 lrwxrwxrwx 1 root root 17 Aug 12 14:51 /dev/drbd/by-disk/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../../../drbd3 42756 0 lrwxrwxrwx 1 root root 17 Aug 11 14:00 /dev/drbd/by-disk/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2XX -> ../../../../drbd2 63889 0 lrwxrwxrwx 1 root root 9 Aug 12 13:42 /dev/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../sdb 7429 0 lrwxrwxrwx 1 root root 9 Aug 10 16:45 /dev/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2XX -> ../../sdd
root@toy02:~# hdparm -I /dev/sdb| grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2AXroot@toy02:~# hdparm -I /dev/sdd| grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2XXroot@toy02:~# hdparm -I /dev/sde| grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2AX/dev/sdb and /dev/sde have the same serial number!
But there are really only 2 HGST drives in the server (and some other seagate disks, non-relevant here).
root@toy02:~# find /dev -ls | grep sde 10391 0 brw-rw---- 1 root disk Aug 10 16:45 /dev/sde 8360 0 lrwxrwxrwx 1 root root 9 Aug 10 16:45 /dev/disk/by-path/pci-0000:08:00.0-scsi-0:1:2:0 -> ../../sde 8355 0 lrwxrwxrwx 1 root root 6 Aug 10 16:45 /dev/block/8:64 -> ../sde
root@toy02:~# find /dev -ls | grep sdb 10382 0 brw-rw---- 1 root disk Aug 12 13:42 /dev/sdb 68794 0 lrwxrwxrwx 1 root root 9 Aug 12 13:42 /dev/disk/by-uuid/411af13f-6cae-4f03-99dc-5941acb3135b -> ../../sdb 12410 0 lrwxrwxrwx 1 root root 9 Aug 12 13:42 /dev/disk/by-path/pci-0000:08:00.0-sas-0x1221000002000000-lun-0 -> ../../sdb 68791 0 lrwxrwxrwx 1 root root 9 Aug 12 13:42 /dev/disk/by-label/data -> ../../sdb 63890 0 lrwxrwxrwx 1 root root 9 Aug 12 13:42 /dev/disk/by-id/wwn-0x5000cca24ec137db -> ../../sdb 63889 0 lrwxrwxrwx 1 root root 9 Aug 12 13:42 /dev/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../sdb 12403 0 lrwxrwxrwx 1 root root 6 Aug 12 13:42 /dev/block/8:16 -> ../sdb
/dev/sdb and /dev/sde are in reality the same physical disk!
Signature · 6 lines
Ullrich Horlacher Informationssysteme und Serverbetrieb IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Quoted 1 line · Ulli Horlacher
/dev/sdband/dev/sdeare in reality the same physical disk!
When does all of this confusion happen? Is it already confused before mkfs or only after mkfs or only after mount? I would find out what instigates it, wipe all signatures from everything, reboot, start from scratch, and then strace the command that causes the confusion. And attach that output as well as the entire dmesg to a bug report. Just my 2 cents, I have no idea what's going on but sounds like a block layer and/or drdb bug that's triggered by Btrfs multiple device setups.
Signature · 1 line
Chris Murphy
[adding Ulli back into the cc list]
Quoted 5 lines · Chris Murphy
Quoted 1 line · Ulli Horlacher
/dev/sdband/dev/sdeare in reality the same physical disk!
Quoted 8 lines
When does all of this confusion happen? Is it already confused before mkfs or only after mkfs or only after mount? I would find out what instigates it, wipe all signatures from everything, reboot, start from scratch, and then strace the command that causes the confusion. And attach that output as well as the entire dmesg to a bug report. Just my 2 cents, I have no idea what's going on but sounds like a block layer and/or drdb bug that's triggered by Btrfs multiple device setups.
If (some of) the DRBD host devices are also physically present on
the machine to which the DRBDs are exported, then you're in the same situation as having block-level snapshots or dd copies of the data -- the FS will see two devices (the backing store and the DRBD) which are have the same UUID. It will pick an arbitrary one to write to, which is probably not something that the DRBD driver will cope with very well, I suspect.
I think the solution here would be to blacklist the backing store
from btrfs dev scan. I recall that there was such a capability at some point -- I don't know if it made it into the userspace tools?
Hugo.
Signature · 4 lines
Hugo Mills | Great oxymorons of the world, no. 7: hugo@... carfax.org.uk | The Simple Truth http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 25 lines · Hugo Mills
[adding Ulli back into the cc list]
Quoted 5 lines · Chris Murphy
Quoted 1 line · Ulli Horlacher
/dev/sdband/dev/sdeare in reality the same physical disk!Quoted 8 lines
When does all of this confusion happen? Is it already confused before mkfs or only after mkfs or only after mount? I would find out what instigates it, wipe all signatures from everything, reboot, start from scratch, and then strace the command that causes the confusion. And attach that output as well as the entire dmesg to a bug report. Just my 2 cents, I have no idea what's going on but sounds like a block layer and/or drdb bug that's triggered by Btrfs multiple device setups.
If (some of) the DRBD host devices are also physically present on
the machine to which the DRBDs are exported, then you're in the same situation as having block-level snapshots or dd copies of the data -- the FS will see two devices (the backing store and the DRBD) which are have the same UUID. It will pick an arbitrary one to write to, which is probably not something that the DRBD driver will cope with very well, I suspect.
Quoted 3 lines
I think the solution here would be to blacklist the backing store
from btrfs dev scan. I recall that there was such a capability at some point -- I don't know if it made it into the userspace tools?
That makes sense. But then this would also affect XFS also I'd think, except XFS will refuse to mount if the kernel sees two of the same fs UUID.
Signature · 1 line
Chris Murphy
Quoted 7 lines · Chris Murphy
Quoted 1 line · Ulli Horlacher
/dev/sdband/dev/sdeare in reality the same physical disk!When does all of this confusion happen? Is it already confused before mkfs or only after mkfs or only after mount?
I have not looked closely before the mkfs.btrfs, because I noticed no problems.
Quoted 2 lines
I would find out what instigates it, wipe all signatures from everything, reboot, start from scratch
This is not an option for me, because this server (despite its name) is a production system.
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<CAJCQCtRTvn-Xu_ipM7pCTtVuxUm0m7kjqt=F=+q6cj0vOOhF7g@mail.gmail.com>
Quoted 12 lines
root@toy02:~# df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdb btrfs 3906909856 140031696 3765056176 4% /dataroot@toy02:~# btrfs filesystem show /data Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 129.81GiB devid 3 size 1.82TiB used 67.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 67.03GiB path /dev/sdb
Quoted 3 lines
Btrfs v3.12
==> btrfs shows the wrong (raw) device
/dev/sdbinstead of/dev/drbd3!
Don't be too alarmed by that, progs do a bit of user land fabrication (wrong). kernel may /may-not be using sdb. try -m option.
just in case if it didn't, Then use mount -o devices / btrfs dev scan <dev> option to provide the desired dev path to the kernel.
Thanks, Anand
Quoted 10 lines · anand jain
Quoted 10 lines
root@toy02:~# df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdb btrfs 3906909856 140031696 3765056176 4% /dataroot@toy02:~# btrfs filesystem show /data Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 129.81GiB devid 3 size 1.82TiB used 67.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 67.03GiB path /dev/sdb
Quoted 6 lines
Quoted 3 lines
Btrfs v3.12
==> btrfs shows the wrong (raw) device
/dev/sdbinstead of/dev/drbd3!Don't be too alarmed by that, progs do a bit of user land fabrication (wrong). kernel may /may-not be using sdb. try -m option.
It is really weird: meanwhile (without any mount change or even reboot) I get:
root@toy02:~# btrfs filesystem show
Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b
Total devices 2 FS bytes used 106.51GiB
devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2
devid 4 size 1.82TiB used 82.03GiB path /dev/drbd3==> no more /dev/sdb !
BUT:
root@toy02:~# df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdb btrfs 3906909856 111822636 3793208212 3% /data
root@toy02:~# mount | grep /data /dev/sdb on /data type btrfs (rw)
root@toy02:~# grep /data /proc/mounts /dev/drbd2 /data btrfs rw,relatime,space_cache 0 0
And still, Linux sees 3 HGST devices (there are real 2 drives):
root@toy02:~# hdparm -I /dev/sdb | grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2AXroot@toy02:~# hdparm -I /dev/sdd | grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2XXroot@toy02:~# hdparm -I /dev/sde | grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2AXSignature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Quoted 19 lines · Ulli Horlacher
Quoted 10 lines · anand jain
Quoted 10 lines
root@toy02:~# df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdb btrfs 3906909856 140031696 3765056176 4% /dataroot@toy02:~# btrfs filesystem show /data Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 129.81GiB devid 3 size 1.82TiB used 67.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 67.03GiB path /dev/sdbQuoted 6 lines
Quoted 3 lines
Btrfs v3.12
==> btrfs shows the wrong (raw) device
/dev/sdbinstead of/dev/drbd3!Don't be too alarmed by that, progs do a bit of user land fabrication (wrong). kernel may /may-not be using sdb. try -m option.
Quoted 8 lines
It is really weird: meanwhile (without any mount change or even reboot) I get:
root@toy02:~# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 106.51GiB devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 82.03GiB path /dev/drbd3
And now, after a reboot:
root@toy02:~/bin# btrfs filesystem show
Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b
Total devices 2 FS bytes used 119.82GiB
devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2
devid 4 size 1.82TiB used 82.03GiB path /dev/sdeGRMPF!
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Quoted 31 lines · Ulli Horlacher
Quoted 19 lines · Ulli Horlacher
Quoted 10 lines · anand jain
Quoted 10 lines
root@toy02:~# df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdb btrfs 3906909856 140031696 3765056176 4% /dataroot@toy02:~# btrfs filesystem show /data Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 129.81GiB devid 3 size 1.82TiB used 67.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 67.03GiB path /dev/sdbQuoted 6 lines
Quoted 3 lines
Btrfs v3.12
==> btrfs shows the wrong (raw) device
/dev/sdbinstead of/dev/drbd3!Don't be too alarmed by that, progs do a bit of user land fabrication (wrong). kernel may /may-not be using sdb. try -m option.
Quoted 8 lines
It is really weird: meanwhile (without any mount change or even reboot) I get:
root@toy02:~# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 106.51GiB devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 82.03GiB path /dev/drbd3And now, after a reboot:
Quoted 7 lines
root@toy02:~/bin# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 119.82GiB devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 82.03GiB path /dev/sdeGRMPF!
pls use 'btrfs fi show -m' and just ignore no option or -d if fs is mounted, as -m reads from the kernel.
at mount you could assemble correct set of devices using mount -o device option.
Thanks, -Anand
Quoted 14 lines · Anand Jain
Quoted 14 lines
Quoted 5 lines
root@toy02:~# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 106.51GiB devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 82.03GiB path /dev/drbd3And now, after a reboot:
root@toy02:~/bin# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 119.82GiB devid 3 size 1.82TiB used 82.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 82.03GiB path /dev/sde
Quoted 4 lines
Quoted 1 line
GRMPF!
pls use 'btrfs fi show -m' and just ignore no option or -d if fs is mounted, as -m reads from the kernel.
There is now a new behaviour: after the btrfs mount, I can see shortly the wrong raw device /dev/sde and a few seconds later there is the correct /dev/drbd3 :
root@toy02:/etc# umount /data
root@toy02:/etc# mount /data
root@toy02:/etc# btrfs filesystem show
Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b
Total devices 2 FS bytes used 109.56GiB
devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2
devid 4 size 1.82TiB used 63.03GiB path /dev/sdeBtrfs v3.12
root@toy02:/etc# btrfs filesystem show
Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b
Total devices 2 FS bytes used 109.56GiB
devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2
devid 4 size 1.82TiB used 63.03GiB path /dev/drbd3Btrfs v3.12
root@toy02:/etc# btrfs filesystem show -m
Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b
Total devices 2 FS bytes used 109.56GiB
devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2
devid 4 size 1.82TiB used 63.03GiB path /dev/drbd3Btrfs v3.12
Still, the kernel sees 3 instead of (really) 2 HGST drives:
root@toy02:/etc# hdparm -I /dev/sdb | grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2AXroot@toy02:/etc# hdparm -I /dev/sde | grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2AXroot@toy02:/etc# hdparm -I /dev/sdd | grep Number:
Model Number: HGST HUS724020ALA640
Serial Number: PN2134P5G2P2XXSignature · 6 lines
Ullrich Horlacher Informationssysteme und Serverbetrieb IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<[email protected]>
First of all there is a known issue in handling multiple paths / instances of the same device image in btrfs. Fixing this caused regression earlier. And my survey
[survey] BTRFS_IOC_DEVICES_READY return status
almost told me not to fix the bug.
But these are just a reporting issue which would confuse users, should be fixed.
Quoted 3 lines
There is now a new behaviour: after the btrfs mount, I can see shortly the wrong raw device
/dev/sdeand a few seconds later there is the correct/dev/drbd3:
yep possible. but it does not mean that btrfs kernel is using the new path its just a reporting (bug).
(pls use -m option)
Quoted 31 lines
root@toy02:/etc# umount /data root@toy02:/etc# mount /data root@toy02:/etc# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 109.56GiB devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 63.03GiB path /dev/sdeBtrfs v3.12
root@toy02:/etc# btrfs filesystem show Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 109.56GiB devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 63.03GiB path /dev/drbd3Btrfs v3.12
root@toy02:/etc# btrfs filesystem show -m Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 109.56GiB devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 63.03GiB path /dev/drbd3Btrfs v3.12
Still, the kernel sees 3 instead of (really) 2 HGST drives:
Quoted 7 lines
root@toy02:/etc# hdparm -I /dev/sdb | grep Number: Model Number: HGST HUS724020ALA640 Serial Number: PN2134P5G2P2AXroot@toy02:/etc# hdparm -I /dev/sde | grep Number: Model Number: HGST HUS724020ALA640 Serial Number: PN2134P5G2P2AX
This is important to know but not a btrfs issue. Do you have multiple host paths reaching this this device with serial # PN2134P5G2P2AX ?
Quoted 5 lines · Anand Jain
First of all there is a known issue in handling multiple paths / instances of the same device image in btrfs. Fixing this caused regression earlier. And my survey
[survey]BTRFS_IOC_DEVICES_READYreturn status
almost told me not to fix the bug.
I have subscribed to this list this week, I am a newbie :-)
Quoted 6 lines
Quoted 3 lines
There is now a new behaviour: after the btrfs mount, I can see shortly the wrong raw device
/dev/sdeand a few seconds later there is the correct/dev/drbd3:yep possible. but it does not mean that btrfs kernel is using the new path its just a reporting (bug).
What is the reporting bug: /dev/sde or /dev/drbd3?
Quoted 13 lines
Quoted 13 lines
root@toy02:/etc# btrfs filesystem show -m Label: data uuid: 411af13f-6cae-4f03-99dc-5941acb3135b Total devices 2 FS bytes used 109.56GiB devid 3 size 1.82TiB used 63.03GiB path /dev/drbd2 devid 4 size 1.82TiB used 63.03GiB path /dev/drbd3Still, the kernel sees 3 instead of (really) 2 HGST drives:
root@toy02:/etc# hdparm -I /dev/sdb | grep Number: Model Number: HGST HUS724020ALA640 Serial Number: PN2134P5G2P2AX
Quoted 6 lines
Quoted 3 lines
root@toy02:/etc# hdparm -I /dev/sde | grep Number: Model Number: HGST HUS724020ALA640 Serial Number: PN2134P5G2P2AXThis is important to know but not a btrfs issue. Do you have multiple host paths reaching this this device with serial # PN2134P5G2P2AX ?
root@toy02:~# find /dev -ls | grep PN2134P5G2P2AX 14354 0 lrwxrwxrwx 1 root root 17 Aug 14 09:00 /dev/drbd/by-disk/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../../../drbd3 13640 0 lrwxrwxrwx 1 root root 9 Aug 13 16:25 /dev/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../sdb
root@toy02:~# find /dev -ls | grep sdb 7417 0 brw-rw---- 1 root disk Aug 13 16:25 /dev/sdb 12366 0 lrwxrwxrwx 1 root root 9 Aug 13 16:25 /dev/disk/by-path/pci-0000:08:00.0-sas-0x1221000002000000-lun-0 -> ../../sdb 13641 0 lrwxrwxrwx 1 root root 9 Aug 13 16:25 /dev/disk/by-id/wwn-0x5000cca24ec137db -> ../../sdb 13640 0 lrwxrwxrwx 1 root root 9 Aug 13 16:25 /dev/disk/by-id/ata-HGST_HUS724020ALA640_PN2134P5G2P2AX -> ../../sdb 12356 0 lrwxrwxrwx 1 root root 6 Aug 13 16:25 /dev/block/8:16 -> ../sdb
root@toy02:~# find /dev -ls | grep sde 13353 0 brw-rw---- 1 root disk Aug 13 16:24 /dev/sde 15725 0 lrwxrwxrwx 1 root root 9 Aug 13 16:25 /dev/disk/by-uuid/411af13f-6cae-4f03-99dc-5941acb3135b -> ../../sde 15724 0 lrwxrwxrwx 1 root root 9 Aug 13 16:25 /dev/disk/by-label/data -> ../../sde 9394 0 lrwxrwxrwx 1 root root 9 Aug 13 16:24 /dev/disk/by-path/pci-0000:08:00.0-scsi-0:1:2:0 -> ../../sde 9387 0 lrwxrwxrwx 1 root root 6 Aug 13 16:24 /dev/block/8:64 -> ../sde
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<[email protected]>
A couple of observations:
1. BTRFS currently has no knowledge of multipath or anything like that.
In theory it should work fine as long as the multiple device instances
all point to the same storage directly (including having identical block addresses), but we still need to add proper handling for it.
2. Be _VERY_ careful using BTRFS on top of _ANY_ kind of shared storage.
Most non-clustered filesystems will have issues if multiply mounted,
but in almost all cases I've personally seen, it _WILL_ cause irreparable damage to a BTRFS filesystem (we really need to do something like ext4's MMP in BTRFS).
3. See the warnings about doing block level copies and LVM snapshots of BTRFS volumes, the same applies to using it on DRBD currently as well (with the possible exception of remote DRBD nodes (ie, ones without a local copy of the backing store) (in this case, we need to blacklist backing devices for stacked storage (I think the same issue may be present with BTRFS on a MD based RAID1 set).
Quoted 5 lines · Austin S Hemmelgarn
2. Be _VERY_ careful using BTRFS on top of _ANY_ kind of shared storage. Most non-clustered filesystems will have issues if multiply mounted,
but in almost all cases I've personally seen, it _WILL_ cause irreparable damage to a BTRFS filesystem (we really need to do something like ext4's MMP in BTRFS).
Same with ZFS: it has also no MMP and when you mount a shared block device twice you will destroy it irreparable. Kids, do not try this at home - I have done so ;-)
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum IZUS/TIK E-Mail: [email protected] Universitaet Stuttgart Tel: ++49-711-68565868 Allmandring 30a Fax: ++49-711-682357 70550 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Quoted 6 lines · Austin S Hemmelgarn
3. See the warnings about doing block level copies and LVM snapshots of BTRFS volumes, the same applies to using it on DRBD currently as well (with the possible exception of remote DRBD nodes (ie, ones without a local copy of the backing store) (in this case, we need to blacklist backing devices for stacked storage (I think the same issue may be present with BTRFS on a MD based RAID1 set).
I've been using BTRFS on top of DRBD for several years now, what specifically am I meant to avoid?
I have 6 drives mirrored across a local network, this is done with DRBD. At any one time only a single server has the 6 drives mounted with btrfs. Is this a ticking time bomb?
Signature · 3 lines
Gareth Pye - blog.cerberos.id.au Level 2 MTG Judge, Melbourne, Australia "Dear God, I would like to file a bug report"
Quoted 10 lines · Gareth Pye
Quoted 6 lines · Austin S Hemmelgarn
3. See the warnings about doing block level copies and LVM snapshots of BTRFS volumes, the same applies to using it on DRBD currently as well (with the possible exception of remote DRBD nodes (ie, ones without a local copy of the backing store) (in this case, we need to blacklist backing devices for stacked storage (I think the same issue may be present with BTRFS on a MD based RAID1 set).
Quoted 6 lines
I've been using BTRFS on top of DRBD for several years now, what specifically am I meant to avoid?
I have 6 drives mirrored across a local network, this is done with DRBD. At any one time only a single server has the 6 drives mounted with btrfs. Is this a ticking time bomb?
There are two things which are potentially worrisome here:
- Having the same filesystem mounted on more than one machine at a time (which you're not doing).
- Having one or more of the DRBD backing store devices present on the same machine that the DRBD filesystem is mounted on (which you may be doing).
Of these, the first is definitely going to be dangerous. The second
may or may not be, depending on how well DRBD copes with direct writes to its backing store, and how lucky you are about the kernel identifying the right devices to use for the FS.
Hugo.
Signature · 4 lines
Hugo Mills | "Big data" doesn't just mean increasing the font hugo@... carfax.org.uk | size. http://carfax.org.uk/ | PGP: E2AB1DE4 |
I would have been surprised if any generic file system copes well with being mounted in several locations at once, DRBD appears to fight really hard to avoid that happening :)
And yeah I'm doing the second thing, I've successfully switched which of the servers is active a few times with no ill effect (I would expect scrub to give me some significant warnings if one of the disks was a couple of months out of date) so I'm presuming that DRBD copes reasonably well or I've been very lucky. Either that luck is very deterministic, DRBD copes correctly, or I've been very very lucky.
Very very lucky doesn't sound likely.
Quoted 39 lines · Hugo Mills
Quoted 10 lines · Gareth Pye
Quoted 6 lines · Austin S Hemmelgarn
3. See the warnings about doing block level copies and LVM snapshots of BTRFS volumes, the same applies to using it on DRBD currently as well (with the possible exception of remote DRBD nodes (ie, ones without a local copy of the backing store) (in this case, we need to blacklist backing devices for stacked storage (I think the same issue may be present with BTRFS on a MD based RAID1 set).
Quoted 6 lines
I've been using BTRFS on top of DRBD for several years now, what specifically am I meant to avoid?
I have 6 drives mirrored across a local network, this is done with DRBD. At any one time only a single server has the 6 drives mounted with btrfs. Is this a ticking time bomb?
There are two things which are potentially worrisome here:
- Having the same filesystem mounted on more than one machine at a time (which you're not doing).
- Having one or more of the DRBD backing store devices present on the same machine that the DRBD filesystem is mounted on (which you may be doing).
Of these, the first is definitely going to be dangerous. The second
may or may not be, depending on how well DRBD copes with direct writes to its backing store, and how lucky you are about the kernel identifying the right devices to use for the FS.Hugo.
--
Hugo Mills | "Big data" doesn't just mean increasing the font hugo@... carfax.org.uk | size. http://carfax.org.uk/ | PGP: E2AB1DE4 |
Signature · 3 lines
Gareth Pye - blog.cerberos.id.au Level 2 MTG Judge, Melbourne, Australia "Dear God, I would like to file a bug report"
Quoted 4 lines · Gareth Pye
I would have been surprised if any generic file system copes well with being mounted in several locations at once, DRBD appears to fight really hard to avoid that happening :)
Quoted 9 lines
And yeah I'm doing the second thing, I've successfully switched which of the servers is active a few times with no ill effect (I would expect scrub to give me some significant warnings if one of the disks was a couple of months out of date) so I'm presuming that DRBD copes reasonably well or I've been very lucky. Either that luck is very deterministic, DRBD copes correctly, or I've been very very lucky.
Very very lucky doesn't sound likely.
Yeah, I'd be willing to bet that DRBD does cope well with direct writes to the backing store (either that or it prevents the kernel from doing that, which would be even better and would not surprise me at all). In my experience it's one of the most resilient shared storage options out there.