Skip to contentSkip to conversation

"--re-add for /dev/sdb1 to /dev/md0 is not possible"

DISCUSSION Andy Smith 3 participants 6 messages Sep 15, 2020 – Nov 28, 2020 MD RAID

lore
Share & tools
lore

Andy Smith

author
#0 lore raw

Hi,

In my continuing goal to remove the bad blocks log from any of my arrays and not have one on any new arrays I create, I wrote this article:

https://strugglers.net/~andy/blog/2020/09/13/debian-installer-mdadm-configuration-and-the-bad-blocks-controversy

Shortly afterwards someone on Hacker News¹ said that it is possible to remove the BBL by failing and re-adding devices, like so:

# mdadm /dev/md127 --fail /dev/sda --remove /dev/sda --re-add /dev/sda --update=no-bbl

I tried that on Ubuntu 18.04:

$ mdadm --version
mdadm - v4.1-rc1 - 2018-03-22
$ sudo mdadm --fail /dev/md0 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1 --update=no-bbl
mdadm: set /dev/sdb1 faulty in /dev/md0
mdadm: hot removed /dev/sdb1 from /dev/md0
mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible
$ sudo mdadm --add /dev/md0 /dev/sdb1 --update=no-bbl
mdadm: --update in Manage mode only allowed with --re-add.
$ sudo mdadm --add /dev/md0 /dev/sdb1
mdadm: added /dev/sdb1
$ sudo mdadm --examine-badblocks /dev/sdb1
Bad-blocks list is empty in /dev/sdb1

I tried it on Debian buster:

$ mdadm --version
mdadm - v4.1 - 2018-10-01
$ sudo mdadm --fail /dev/md6 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1 --update=no-bbl
mdadm: set /dev/sdb1 faulty in /dev/md6
mdadm: hot removed /dev/sdb1 from /dev/md6
mdadm: --re-add for /dev/sdb1 to /dev/md6 is not possible

So, is that supposed to work and if so, why doesn't it work for me?

In both cases these are simple two device RAID-1 metadata version 1.2 arrays. Neither has bitmaps.

Thanks,
Andy

¹ https://news.ycombinator.com/item?id=24479235

author
#1 lore raw
Quoted 6 lines · Andy Smith

mdadm: --re-add for /dev/sdb1 to /dev/md6 is not possible

So, is that supposed to work and if so, why doesn't it work for me?

In both cases these are simple two device RAID-1 metadata version 1.2 arrays. Neither has bitmaps.

The lack of bitmaps was why I couldn't use --re-add. I was testing with small arrays that didn't get a bitmap by default. As mentioned on the wiki, a bitmap can be added like:

# mdadm --grow --bitmap=internal /dev/mdX

So, as pointed out to me by Jakub Wilk, it is possible to remove the bad blocks log on a running array by failing and re-adding each device in turn, as long as you are willing to experience lower / no redundancy while it rebuilds.

# mdadm --fail /dev/md0 /dev/sdb1 \
        --remove /dev/sdb1 \
        --re-add /dev/sdb1 \
        --update=no-bbl

Cheers,
Andy

#2 lore raw
In reply to Andy Smith
Quoted 38 lines

In my continuing goal to remove the bad blocks log from any of my arrays and not have one on any new arrays I create, I wrote this article:

https://strugglers.net/~andy/blog/2020/09/13/debian-installer-mdadm-configuration-and-the-bad-blocks-controversy

Shortly afterwards someone on Hacker News¹ said that it is possible to remove the BBL by failing and re-adding devices, like so:

# mdadm /dev/md127 --fail /dev/sda --remove /dev/sda --re-add /dev/sda
--update=no-bbl

I tried that on Ubuntu 18.04:

$ mdadm --version
mdadm - v4.1-rc1 - 2018-03-22
$ sudo mdadm --fail /dev/md0 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1
--update=no-bbl
mdadm: set /dev/sdb1 faulty in /dev/md0
mdadm: hot removed /dev/sdb1 from /dev/md0
mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible
$ sudo mdadm --add /dev/md0 /dev/sdb1 --update=no-bbl
mdadm: --update in Manage mode only allowed with --re-add.
$ sudo mdadm --add /dev/md0 /dev/sdb1
mdadm: added /dev/sdb1
$ sudo mdadm --examine-badblocks /dev/sdb1
Bad-blocks list is empty in /dev/sdb1

I tried it on Debian buster:

$ mdadm --version
mdadm - v4.1 - 2018-10-01
$ sudo mdadm --fail /dev/md6 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1
--update=no-bbl
mdadm: set /dev/sdb1 faulty in /dev/md6
mdadm: hot removed /dev/sdb1 from /dev/md6
mdadm: --re-add for /dev/sdb1 to /dev/md6 is not possible
Quoted 4 lines

So, is that supposed to work and if so, why doesn't it work for me?

In both cases these are simple two device RAID-1 metadata version 1.2 arrays. Neither has bitmaps.

I've tried this as well, and ran

mddev=/dev/md0
diskdev=/dev/sdm

mdadm --fail $mddev $diskdev
mdadm --remove $mddev $diskdev
mdadm --re-add $mddev $diskdev

All returned ok, but sdm was listed as a slave, thus not active anymore. After this, a new --remove and --re-add didn't work, so it'll take some hours to --add it again (yet another time).

It would be very nice if someone at linux-raid could prioritise this rather obvious bug in the bbl code, where the bbl keeps replicating itself over and over, regardless of any actual failures on the disks. IMHO the whole BBL should be scrapped, as mentioned earlier, since it really has no function. Mapping out bad sectors is for the drive to decide and if it can't handle it, it should be kicked out of the array.

Vennlig hilsen

roy

Signature · 6 lines
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.
#3 lore raw
Quoted 1 line · Roy Sigurd Karlsbakk

It would be very nice if someone at linux-raid could prioritise this rather obvious bug in the bbl code, where the bbl keeps replicating itself over and over, regardless of any actual failures on the disks. IMHO the whole BBL should be scrapped, as mentioned earlier, since it really has no function. Mapping out bad sectors is for the drive to decide and if it can't handle it, it should be kicked out of the array.

Well, I guess if you want a volunteer, you're it! Thanks for steping up to the plate!

Cheers,
Wol

author
#5 lore raw
In reply to Roy Sigurd Karlsbakk

Hi Roy,

Quoted 1 line · Roy Sigurd Karlsbakk

Will anyone look into this issue, please? It really is a problem.

You already did ask once and got the response to get on with it and let everyone else know; asking further times isn't going to get anywhere so I suggest you do what is within your power to do.

If the option of coding the fix yourself is not available and you can't convince (or pay?) someone else to do it, I guess that just leaves you with the option of disabling BBL.

At least we did go through this before and established that it's possible to do without downtime, though with short period of loss of redundancy.

Cheers,
Andy

#4 lore raw
In reply to antlists

Will anyone look into this issue, please? It really is a problem.

Vennlig hilsen

roy

Signature · 8 lines
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

----- Original Message -----
Quoted 19 lines

From: "Wols Lists" <[email protected]>
To: "Roy Sigurd Karlsbakk" <[email protected]>, "Andy Smith" <[email protected]>
Cc: "Linux Raid" <[email protected]>
Sent: Wednesday, 23 September, 2020 01:18:26
Subject: Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

Quoted 6 lines · Roy Sigurd Karlsbakk

It would be very nice if someone at linux-raid could prioritise this rather obvious bug in the bbl code, where the bbl keeps replicating itself over and over, regardless of any actual failures on the disks. IMHO the whole BBL should be scrapped, as mentioned earlier, since it really has no function. Mapping out bad sectors is for the drive to decide and if it can't handle it, it should be kicked out of the array.

Well, I guess if you want a volunteer, you're it! Thanks for steping up to the plate!

Cheers,
Wol