delete missing /dev/sdd which is now added as /dev/sdd1

I have a BTRFS mirrored raid on three disks: 500 GB, 1000 GB, 1500 GB. The 1500 GB is going bad.

I bought a 4000 GB and wanted to make a 1500 GB and also a 2500 GB partition. By mistake I made a 1.5 GB and did not realize it until much later.

I added the 1.5 GB to my raid and tried removing the 1500 GB, I obviously coudn't. I ended up turning off the machine and unplugging the 1500 GB and tried the btrfs delete missing, but that failed too because delete missing needs new space available otherwise it will fail.

So I reformatted my 1500 GB and added it to the raid again, it showed up... but twice... I had one empty /dev/sdd and one non-empty /dev/sdd.

I figured I could fix it by removing and formatting it again, this time creating a partition using fdisk, it would change the device to /dev/sdd1.

I added the /dev/sdd1 to my raid and deleted the missing. Now the /dev/sdd does not show up anymore and I have a /dev/sdd1. But I still have the **** Some devices missing and the command btrfs delete missing /mnt does not actually do anything. I tried btrfs delete /dev/sdd /mnt but it said that /dev/sdd was busy.

How can I tell btrfs that it is NOT the same drive?

Here is a screenshot from the machine:
https://www.dropbox.com/s/f33l4ncsni7wd0s/2014-08-13%2007.31.42.jpg

I figured I could somehow change the /dev/ name, but I am not sure how.

Rasmus

Have you tried
btrfs device delete missing /mnt/mountpoint ?

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

/mnt is the mount point. I now did the following

dd if=/dev/zero of=/dev/sdd

Finally, btrfs does not recognize the partition as part of my raid anymore, but what's worse is that it now tells me that two missing devices is a no go, so I cannot mount it as anything but read-only. All the data is there, /dev/sdd1 was never used and /dev/sdd was the originally missing one. Mounting as read-only prevents you from adding devices.

I guess I have to start over... At least the old pool is still readable. Sucks that I have to rebuild it though. I guess there is no way to keep my snapshots without having them fill their full size.

Rasmus Abrahamsen posted on Wed, 13 Aug 2014 17:36:06 +0200 as excerpted:

There's a patch in-flight that should, I believe, help with this. I haven't followed the specific details, but IIRC, it has something to do with a missing sysfs entry. Because of that missing entry, btrfs (userspace) doesn't have the same view of filesystem device status that the kernel has, and device add/replace/delete can go haywire. Additionally, in some cases it triggers too many missing devices errors, forcing the mount read-only, which in turn won't let you add devices to fix the problem even if due to the circumstances you know all files are actually there and intact.

So at this point you have three choices:

1) Give up on the filesystem, do a mkfs and go from there.

2) Wait for the patch to make it into kernel 3.17 and go from there.

3) Go read the list and find and cherrypick that patch, then rebuild with it included, and go from there.

Meanwhile, an observation on the btrfs behavior that contributed to getting you into this situation in the first place, and how to avoid it in the future:

Btrfs filesystem and filesystem device detection relies on a "magic string" found in the superblock to identify btrfs filesystems, and other metadata within the superblock to determine what specific btrfs filesystem it belongs to. When events such as btrfs device add/delete/ replace that change the devices composing a filesystem don't complete properly, this superblock information can remain to confuse btrfs when it tries working with the remaining device components in the filesystem.

That's what was happening to you, tho the missing sysfs entries mentioned above make the situation rather worse. Btrfs was still detecting these remaining bits of superblock information saying that the device in question belonged to the btrfs it originated with, and was getting confused.

One way to avoid this problem is to use a tool called wipefs, part of util-linux, to erase the magic-string btrfs detection uses, so that particular device will no longer be detected as part of a btrfs, and thus can't confuse btrfs. Alternatively, you can simply zero-out the correct 8 bytes on the raw device manually, using for instance dd to write directly to the device.

See this link from the btrfs wiki problem-FAQ for more (including how to restore the 8-byte magic string using dd if you decide you made a mistake, something not really possible if you zero out the entire device as you did above):

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_to_clean_up_old_superblock_.3F

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Based on integration right now it looks like a lot of stuff.

Could be this one:
http://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/patch/?id=0c414ebcd5ba515b58249099619a5d9dbcc1a3b8

Chris Murphy

Wow, such a long and detailed message. Thank you so much for taking the time to do this.

Well, to be honest, I am starting to like the idea of starting over, although it will be a pain.

Besides, I would have to wait for any distro to make a live disk with the new btrfs version because my linux partition is also on that raid.

I also learned that "btrfs delete missing" should only be used if the drive is gone forever, not to force remove it because of a mistake I did myself (tried to replace it with the 1,5 GB partition).