“Bug”-report: inconsistency kernel <-> tools

Hello,

We had set up a btrfs-fs over 6 hot-plugable SAS-disks for testing and got it into a state where kernel and btrfs-tools do not agree any more about the state of the filesystem.

We do not remember exaclty what we did, but roughly it was something like this (on the running system). THIS IS FROM MEMORY!

(1) pulled out one disk
(2) removed disk from btrfs
(3) rebalanced btrfs
(4) pulled out another disk
(5) removed disk from btrfs
(6) rebalanced btrfs

This went fine sofar.

(7) reinserted disk (and rebooted)
At some point before reboot the first 10 sectors of one disk were zeroed to test if the disk gets removed from the btrfs.

Now btrfs-tools showed:

---------------------------------------------------------------------------

# btrfs fi show                      
failed to read /dev/sr0
Label: 'BTRFS_RAID'  uuid: 807193fd-17de-4088-9a54-3e7cacdc89db
        Total devices 6 FS bytes used 3.07GB
        devid    4 size 931.00GB used 75.00GB path /dev/sdf
        devid    5 size 931.00GB used 324.03GB path /dev/sde
        devid    6 size 931.00GB used 83.03GB path /dev/sdd
        devid    3 size 931.00GB used 326.03GB path /dev/sdc
        devid    2 size 931.00GB used 326.03GB path /dev/sdb
        devid    1 size 931.00GB used 324.04GB path /dev/sda

Btrfs Btrfs v0.19
---------------------------------------------------------------------------

As far as we can tell, only four of the disks are considered part of the btrfs by kernel. There were only four “btrfs: bdev”-lines in dmesg and only four disks took part in balancing. “btrfs device scan” says:

unable to scan the device '/dev/sdd' - Device or resource busy

and balance does not balance theses two devices (of 6)

It was neither possible to remove the disk from the btrfs via “btrfs device delete” nor adding them via “btrfs device add”.

(8) a colleague swaped the two disk

Now btrfs-tools showed:

---------------------------------------------------------------------------

# btrfs fi show
failed to read /dev/sr0
Label: 'BTRFS_RAID'  uuid: 807193fd-17de-4088-9a54-3e7cacdc89db
        Total devices 5 FS bytes used 3.01GB
        devid    6 size 931.00GB used 83.03GB path /dev/sdf
        devid    4 size 931.00GB used 75.00GB path /dev/sdd
        devid    5 size 931.00GB used 325.03GB path /dev/sde
        devid    3 size 931.00GB used 326.03GB path /dev/sdc
        devid    2 size 931.00GB used 325.03GB path /dev/sdb
        devid    1 size 931.51GB used 326.04GB path /dev/sda

Btrfs Btrfs v0.19
---------------------------------------------------------------------------

Claiming the btrfs has 5 disk, but listing 6 disks out of 5 (6 of 5).

He finally managed to get the btrfs complete again by overwriting the first 100G of the two disk. After this the btrfs-tools (correctly) stated a filesystem with 4 disk and it was possible to add the two disk again.

Assumption:
kernel and btrfs do not share the same view of the filesystem.

In this state commands to repair the filesystem do not work, because they are either rejected by the tools or by the kernel.

A tool that allows a disk/partition to be marked as not-a-btrfs-part would be nice.

A “/proc/btrfs” showing the kernels view of the filesystem would be usefull.

MfG
bmg

„Des is völlig wurscht, was heut beschlos- | M G Berberich
 sen wird: I bin sowieso dagegn!“          | [email protected]
(SPD-Stadtrat Kurt Schindler; Regensburg)  | www.fmi.uni-passau.de/~berberic

IIRC the superblock is not placed at the beginning of the disk. On the basis of [1] it should be near the 64KB (around the sector #128)

[1] https://btrfs.wiki.kernel.org/index.php/User:Wtachi/On-disk_Format#Superblock

"btrfs filesystem show" shows the content of the disks, which could be unrelated to the kernel status. Pay attention that if the data is not flushed to the disk the report of "btrfs fi show" could be unreliable.

I posted few days ago a patch which adds the sysfs support to btrfs. With this support it is possible to know the real state of the disks.

For example I have a filesystem with 4 disks (note "Total devices 4"):

  ghigo@emulato:~$ sudo btrfs fi show
  Label: 'btrfs3'  uuid: 2a66286d-63e9-4ed5-b347-5af5e4ada814
	Total devices 4 FS bytes used 284.00KB
	devid    4 size 100.00GB used 8.01GB path /dev/vdj
	devid    3 size 100.00GB used 6.04GB path /dev/vdi
	devid    5 size 100.00GB used 0.00 path /dev/vdh
	devid    1 size 100.00GB used 7.05GB path /dev/vdg

Btrfs Btrfs v0.19

My sysfs interface says that the filesystem is composed by 4 disks:

  ghigo@emulato:~$ cat /sys/fs/btrfs/filesystems/2a66286d-
  63e9-4ed5b347-5af5e4ada814/fs_devices/open_devices
  4

Then I remove 1 disk

  ghigo@emulato:~$ sudo btrfs dev del /dev/vdi  /mnt/btrfs3/

Now the sysfs interface says:

  ghigo@emulato:~$ cat /sys/fs/btrfs/filesystems/2a66286d-
  63e9-4ed5b347-5af5e4ada814/fs_devices/open_devices
  3

But "btrfs filesystem show" says (note still "Total devices 4"):

  ghigo@emulato:~$ sudo btrfs fi show
  failed to read /dev/sr0
  Label: 'btrfs3'  uuid: 2a66286d-63e9-4ed5-b347-5af5e4ada814
	Total devices 4 FS bytes used 92.00KB
	devid    4 size 100.00GB used 7.00GB path /dev/vdj
	devid    3 size 100.00GB used 6.04GB path /dev/vdi
	devid    5 size 100.00GB used 5.06GB path /dev/vdh
	devid    1 size 100.00GB used 6.08GB path /dev/vdg

Btrfs Btrfs v0.19

Then I do a sync

  ghigo@emulato:~$ sync
  ghigo@emulato:~$ sudo btrfs fi show
  failed to read /dev/sr0
  Label: 'btrfs3'  uuid: 2a66286d-63e9-4ed5-b347-5af5e4ada814
	Total devices 3 FS bytes used 92.00KB
	devid    4 size 100.00GB used 7.00GB path /dev/vdj
	devid    3 size 100.00GB used 6.04GB path /dev/vdi
	devid    5 size 100.00GB used 5.06GB path /dev/vdh
	devid    1 size 100.00GB used 6.08GB path /dev/vdg

Btrfs Btrfs v0.19

(note "Total devices 3")

And magically the filesystem is now composed by three disks. However 4 physical devices are show. This because the disk /dev/vdi superblock says that the disk is still valid (after the "btrfs device del" the disk is not touched any more)

In the past Hubert posted a patch [2] to clear a btrfs superblock. A further enhancement of the "btrfs device del" could be to reset automatically the first superblock (leaving the backup ones unaffected).

[2] http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17065

Just for the record, the first is at 64KiB; each subsequent one is
shifted 12 bits left (256MiB, 1TiB, 4EiB, 16ZiB, 64YiB).

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- This chap Anon is writing some perfectly lovely stuff ---      
                             at the moment.                              

I have to correct myself. When a device is removed its superblock is zero-ed (from btrfs_rm_device():

[...]

        /*
         * at this point, the device is zero sized.  We want to
         * remove it from the devices list and zero out the old super
         */
        if (clear_super) {
                /* make sure this device isn't detected as part of
                 * the FS anymore
                 */
                memset(&disk_super->magic, 0, sizeof(disk_super->magic));
                set_buffer_dirty(bh);
                sync_dirty_buffer(bh);
        }

[...]

clear_super is set to true when the device is writeable.

However making a test I found both the behaviours: sometime the removed disk disappears from the output of "btrfs fi show" and sometime not...

May be that there is a bug somewhere...

Hi all, Yan,

I became crazy looking at this bug. I found that a debian package raises the bug, but when I compiled the source by hand the bug disappeared... Finally I discovered that this bug depends by an uninitialized variable; this lead to the unpredictable behaviour.

The problem is that when a device is removed, the function btrfs_read_dev_super() should ignore it. In fact the kernel clear the magic number in the *first* superblock. However the function btrfs_read_dev_super() checks also the backup superblocks and when it found a valid one, the function returns success.

Lukely (?) this function fails very often because the fsid of the backup superblock are checked against an uninitialized buffer. However when this check has success this device is considered suitable even tough it was removed.

The BUG is in the function btrfs_read_dev_super():

int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)

{
        u8 fsid[BTRFS_FSID_SIZE];

[...]

line 933:

        for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
                bytenr = btrfs_sb_offset(i);
                ret = pread64(fd, &buf, sizeof(buf), bytenr);
                if (ret < sizeof(buf))
                        break;

                if (btrfs_super_bytenr(&buf) != bytenr ||
                    strncmp((char *)(&buf.magic), BTRFS_MAGIC,
                            sizeof(buf.magic)))
                        continue;

                if (i == 0)
                        memcpy(fsid, buf.fsid, sizeof(fsid));
                else if (memcmp(fsid, buf.fsid, sizeof(fsid)))
                        continue;

                if (btrfs_super_generation(&buf) > transid) {
                        memcpy(sb, &buf, sizeof(*sb));
                        transid = btrfs_super_generation(&buf);
                }
        }

When a device is removed, the *first* superblock magic field is zeroed so the first check "strncmp((char *)(&buf.magic), BTRFS_MAGIC,..." fails , "i" is increased, and the "continue" statement is execute. Then the check "memcmp(fsid...." became unreliable in the next iteration because the fsid variable is not initialized.

To me the test is unclear: what is the purpose to continue when the superblocks (the original one and its backup) refer to different fsid: there is something wrong which require an user decision... May be that Yan added this check (see commit 50860d6e31c28cf4789ef099729dfbce2108620a ) to converting from different format ? Yan do you remember something about this code ?

The enclosed patch corrects the initialization of the fsid variable; morover if the fsid are different between the superblocks (the original one and its backup) the function fails because the device cannot be trusted. Finally it is handled the special case when the magic fields is zeroed in the *first* superblock. In this case the device is skipped.

BR
G.Baroncelli

Signed-off-by: Goffredo Baroncelli <[email protected]>

index b21a87f..82fc3b8 100644
@@ -910,6 +910,7 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
 int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 

sb_bytenr)
{
u8 fsid[BTRFS_FSID_SIZE];
+ int fsid_is_initialized = 0;
struct btrfs_super_block buf;
int i;
int ret;
@@ -936,15 +937,26 @@ int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)
if (ret < sizeof(buf))
break;

- if (btrfs_super_bytenr(&buf) != bytenr ||
- strncmp((char *)(&buf.magic), BTRFS_MAGIC,
+ if (btrfs_super_bytenr(&buf) != bytenr )
+ continue;
+ /* if magic is NULL, the device was removed */
+ if (buf.magic == 0 && i==0)
+ return -1;
+ if (strncmp((char *)(&buf.magic), BTRFS_MAGIC,
sizeof(buf.magic)))
continue;

- if (i == 0)
+ if (!fsid_is_initialized){
memcpy(fsid, buf.fsid, sizeof(fsid));
- else if (memcmp(fsid, buf.fsid, sizeof(fsid)))
- continue;
+ fsid_is_initialized = 1;
+ } else if (memcmp(fsid, buf.fsid, sizeof(fsid))) {
+ /*
+ * the superblocks (the original one and
+ * its backups) contain data of different
+ * filesystems -> the disk cannot be trusted
+ */
+ return -1;
+ }

if (btrfs_super_generation(&buf) > transid) {
memcpy(sb, &buf, sizeof(*sb));

index b21a87f..82fc3b8 100644
@@ -910,6 +910,7 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
 int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)
 {
 	u8 fsid[BTRFS_FSID_SIZE];
+	int fsid_is_initialized = 0;
 	struct btrfs_super_block buf;
 	int i;
 	int ret;

@@ -936,15 +937,26 @@ int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)
 		if (ret < sizeof(buf))
 			break;
 
-		if (btrfs_super_bytenr(&buf) != bytenr ||
-		    strncmp((char *)(&buf.magic), BTRFS_MAGIC,
+		if (btrfs_super_bytenr(&buf) != bytenr )
+			continue;
+		/* if magic is NULL, the device was removed */
+		if (buf.magic == 0 && i==0)
+			return -1;
+		if (strncmp((char *)(&buf.magic), BTRFS_MAGIC,
 			    sizeof(buf.magic)))
 			continue;
 
-		if (i == 0)
+		if (!fsid_is_initialized){
 			memcpy(fsid, buf.fsid, sizeof(fsid));
-		else if (memcmp(fsid, buf.fsid, sizeof(fsid)))
-			continue;
+			fsid_is_initialized = 1;
+		} else if (memcmp(fsid, buf.fsid, sizeof(fsid))) {
+			/* 
+			 * the superblocks (the original one and
+			 * its backups) contain data of different
+			 * filesystems -> the disk cannot be trusted
+			 */
+			return -1;
+		}
 
 		if (btrfs_super_generation(&buf) > transid) {
 			memcpy(sb, &buf, sizeof(*sb));

This is a repost because I rebased the change. The first attempt was done with the email "[BTRFS-PROGS][BUG][PATCH] Incorrect detection of a removed device [was Re: “Bug”-report: inconsistency kernel <-> tools]" dated 08/31/2012.

In the function btrfs_read_dev_super() it is possible to use the variable fsid without initialisation.

In btrfs_read_dev_super(), during the scan of the superblock the variable fsid is initialised with the value of fsid of the first superblock. But if the first superblock contains an incorrect signature this initialisation is skipped.

int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)

{
         u8 fsid[BTRFS_FSID_SIZE];

[...]

line 933:

         for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
                 bytenr = btrfs_sb_offset(i);
                 ret = pread64(fd, &buf, sizeof(buf), bytenr);
                 if (ret < sizeof(buf))
                         break;

                 if (btrfs_super_bytenr(&buf) != bytenr ||
                     strncmp((char *)(&buf.magic), BTRFS_MAGIC,
                             sizeof(buf.magic)))
                         continue;

                 if (i == 0)
                         memcpy(fsid, buf.fsid, sizeof(fsid));
                 else if (memcmp(fsid, buf.fsid, sizeof(fsid)))
                         continue;

                [...]
         }

This bug could be triggered by the command "btrfs device delete", which zeros the signature of the first superblock.

The enclosed patch corrects the initialisation of the fsid variable; moreover if the fsid are different between the superblocks (the original one and its backups) the function fails because the device cannot be trusted. Finally it is handled the special case when the magic fields is zeroed in the *first* superblock. In this case the device is skipped.

Please apply, thank.

You can pull from
http://cassiopea.homelinux.net/git/btrfs-progs-unstable.git
branch
btrfs_read_dev_super-bug

BR
G.Baroncelli

Signed-off-by: Goffredo Baroncelli <[email protected]>

index b21a87f..82fc3b8 100644
@@ -910,6 +910,7 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
 int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 

sb_bytenr)
{
u8 fsid[BTRFS_FSID_SIZE];
+ int fsid_is_initialized = 0;
struct btrfs_super_block buf;
int i;
int ret;
@@ -936,15 +937,26 @@ int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)
if (ret < sizeof(buf))
break;

- if (btrfs_super_bytenr(&buf) != bytenr ||
- strncmp((char *)(&buf.magic), BTRFS_MAGIC,
+ if (btrfs_super_bytenr(&buf) != bytenr )
+ continue;
+ /* if magic is NULL, the device was removed */
+ if (buf.magic == 0 && i==0)
+ return -1;
+ if (strncmp((char *)(&buf.magic), BTRFS_MAGIC,
sizeof(buf.magic)))
continue;

- if (i == 0)
+ if (!fsid_is_initialized){
memcpy(fsid, buf.fsid, sizeof(fsid));
- else if (memcmp(fsid, buf.fsid, sizeof(fsid)))
- continue;
+ fsid_is_initialized = 1;
+ } else if (memcmp(fsid, buf.fsid, sizeof(fsid))) {
+ /*
+ * the superblocks (the original one and
+ * its backups) contain data of different
+ * filesystems -> the disk cannot be trusted
+ */
+ return -1;
+ }

if (btrfs_super_generation(&buf) > transid) {
memcpy(sb, &buf, sizeof(*sb));

index b21a87f..82fc3b8 100644
@@ -910,6 +910,7 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
 int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)
 {
 	u8 fsid[BTRFS_FSID_SIZE];
+	int fsid_is_initialized = 0;
 	struct btrfs_super_block buf;
 	int i;
 	int ret;

@@ -936,15 +937,26 @@ int btrfs_read_dev_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr)
 		if (ret < sizeof(buf))
 			break;
 
-		if (btrfs_super_bytenr(&buf) != bytenr ||
-		    strncmp((char *)(&buf.magic), BTRFS_MAGIC,
+		if (btrfs_super_bytenr(&buf) != bytenr )
+			continue;
+		/* if magic is NULL, the device was removed */
+		if (buf.magic == 0 && i==0)
+			return -1;
+		if (strncmp((char *)(&buf.magic), BTRFS_MAGIC,
 			    sizeof(buf.magic)))
 			continue;
 
-		if (i == 0)
+		if (!fsid_is_initialized){
 			memcpy(fsid, buf.fsid, sizeof(fsid));
-		else if (memcmp(fsid, buf.fsid, sizeof(fsid)))
-			continue;
+			fsid_is_initialized = 1;
+		} else if (memcmp(fsid, buf.fsid, sizeof(fsid))) {
+			/* 
+			 * the superblocks (the original one and
+			 * its backups) contain data of different
+			 * filesystems -> the disk cannot be trusted
+			 */
+			return -1;
+		}
 
 		if (btrfs_super_generation(&buf) > transid) {
 			memcpy(sb, &buf, sizeof(*sb));