"bitmap" line in /proc/mdstat
Hi all
When I have added a new RAID5 yesterday I saw that it has an additional line (bitmap: 0/30 pages [0KB], 65536KB chunk) in /proc/mdstat for md5 in contrast to md[1234] not having this line:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear]
md5 : active raid5 sdt[4] sds[2] sdr[1] sdi[0]
11720662272 blocks super 1.2 level 5, 128k chunk, algorithm 2
[4/4] [UUUU]
bitmap: 0/30 pages [0KB], 65536KB chunkmd0 : active linear md5[4] md1[0] md4[3] md3[2] md2[1]
33697299328 blocks super 1.2 0k rounding
md4 : active raid5 sdj[5] sdm[4] sdl[2] sdk[1]
5860149888 blocks super 1.2 level 5, 128k chunk, algorithm 2
[4/4] [UUUU]
md3 : active raid5 sdn[0] sdq[4] sdp[2] sdo[1]
5860540032 blocks super 1.2 level 5, 128k chunk, algorithm 2
[4/4] [UUUU]
md2 : active raid5 sde[0] sdh[4] sdg[2] sdf[1]
4395412224 blocks super 1.2 level 5, 128k chunk, algorithm 2
[4/4] [UUUU]
md1 : active raid5 sda[0] sdd[3] sdc[2] sdb[1]
5860540032 blocks super 1.2 level 5, 128k chunk, algorithm 2
[4/4] [UUUU]
unused devices: <none>
What I did yesterday was the following:
# mdadm -C /dev/md5 -c 128 -n4 -l5 /dev/sd[irst]
# mdadm --grow /dev/md0 --add /dev/md5
# xfs_growfs /mnt/media-raid
From what I was reading [1,2] it is useful for faster rebuilds.
But why is it not present for md[1234] as I used the same command to create these RAIDs?
Is it only possible for the last of the linear devices, or is it only used for almost empty devices, or was the bitmap introduced in a recent update? I wonder if I should and could add a bitmap for the other devices as well?
Best regards,
Ramon
[1] https://raid.wiki.kernel.org/index.php/Write-intent_bitmap
[2] https://raid.wiki.kernel.org/index.php/Mdstat#bitmap_line
Quoted 6 lines · Ramon Hofer
From what I was reading [1,2] it is useful for faster rebuilds. But why is it not present for md[1234] as I used the same command to create these RAIDs?
Is it only possible for the last of the linear devices, or is it only used for almost empty devices, or was the bitmap introduced in a recent update?
Bitmaps have been available for a long time, but including them on a new array by default is recent. They are most useful if a device is unexpectedly disconnected but otherwise OK. mdadm --re-add will allow that device to rejoin the array and only the missing data (tracked by the bitmap) will have to be written to sync it up.
Quoted 1 line
I wonder if I should and could add a bitmap for the other devices as well?
There is a performance hit, so it should be considered on a case by case basis.
Phil
Thanks for your answer, Phil.
Quoted 9 lines · Phil Turmel
Quoted 6 lines · Ramon Hofer
From what I was reading [1,2] it is useful for faster rebuilds.
But why is it not present for md[1234] as I used the same command to create these RAIDs?
Is it only possible for the last of the linear devices, or is it only used for almost empty devices, or was the bitmap introduced in a recent update?
Quoted 5 lines
Bitmaps have been available for a long time, but including them on a new array by default is recent. They are most useful if a device is unexpectedly disconnected but otherwise OK. mdadm --re-add will allow that device to rejoin the array and only the missing data (tracked by the bitmap) will have to be written to sync it up.
What could be an unexpected disconnect? If a disc is accidentally removed from a slot?
Quoted 4 lines
Quoted 1 line
I wonder if I should and could add a bitmap for the other devices as well?
There is a performance hit, so it should be considered on a case by case basis.
My uneducated guess would be that I don't really need it.
So I should probably remove the bitmap for the newly created device. Although the performance hit is not really an issue in my case, I guess. How much disk space is used for a 12 TB RAID5 bitmap?
Best regards,
Ramon
Quoted 1 line · Ramon Hofer
How much disk space is used for a 12 TB RAID5 bitmap?
I have remove the bitmap with
# mdadm -G /dev/md5 -b none
Comparing the array sizes from before and afterwards it appears that the internal bitmap does not use any disk space. Since the internal bitmap is part of the metadata this makes total sense.
Still I would imagine if I assume that a bit per chunck is needed and I use 128kB chuncks on the 11720662272 blocks of the 12TB RAID5 this is:
log(11720662272)/log(2) = 34 Bytes
Or am I totally wrong?
Best regards,
Ramon
Quoted 5 lines · Ramon Hofer
Still I would imagine if I assume that a bit per chunck is needed and I use 128kB chuncks on the 11720662272 blocks of the 12TB RAID5 this is:
log(11720662272)/log(2) = 34 BytesOr am I totally wrong?
It's negligible, a few bytes in the metadata somewhere.
The bitmap has its own size (--bitmap-chunk option). It's fine to set this to a large value (128MiB, 1GiB, ...). That might help performance and reduce the bytes required for the bitmap itself (not that it matters).
The main issue is performance impact... it's a tradeoff for additional bitmap updates each write, or long resync time after a power loss or similar event.
If performance is not critical (e.g. it's a NAS, not a database) then you can enable it without worries.
Regards
Andreas Klauer
Thanks for the advice, Andreas.
Quoted 6 lines · Andreas Klauer
The main issue is performance impact... it's a tradeoff for additional bitmap updates each write, or long resync time after a power loss or similar event.
If performance is not critical (e.g. it's a NAS, not a database) then you can enable it without worries.
Yes, it is some sort of NAS.
I also have a database for ownCloud on it but I am the only user.
And I use an USV for that machine so I am also not really concerned about power loss.
Best probably to roll a dice :-)
Regards,
Ramon
Can you quantify the performance decrease and if rebuild time is not as important as performance how does one remove the bitmap?
Thank you,
Peter
-----Original Message-----
From: Phil Turmel [mailto:[email protected]]
Sent: Friday, October 14, 2016 4:53 AM
To: Ramon Hofer; [email protected]
Subject: Re: "bitmap" line in /proc/mdstat
Quoted 6 lines · Ramon Hofer
From what I was reading [1,2] it is useful for faster rebuilds. But why is it not present for md[1234] as I used the same command to create these RAIDs?
Is it only possible for the last of the linear devices, or is it only used for almost empty devices, or was the bitmap introduced in a recent update?
Bitmaps have been available for a long time, but including them on a new array by default is recent. They are most useful if a device is unexpectedly disconnected but otherwise OK. mdadm --re-add will allow that device to rejoin the array and only the missing data (tracked by the bitmap) will have to be written to sync it up.
Quoted 1 line
I wonder if I should and could add a bitmap for the other devices as well?
There is a performance hit, so it should be considered on a case by case basis.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Peter Sangas
Can you quantify the performance decrease and if rebuild time is not as important as performance how does one remove the bitmap?
From the Linux RAID wiki:
"Some limited measurements I have done suggest up to 10% slowdown, though usually less. Possibly some tuning can make it much better." [1]
[1] https://raid.wiki.kernel.org/index.php/Write-intent_bitmap#How_Bitmaps_Work_by_Q.26A
Regards,
Ramon
Thanks Ramon.
Do you know why some of your mds were created with bitmaps and others were not? When I installed Ubuntu 16.04 and used the installer to create a RAID1 some of the mds had bitmaps and some did not.
Thanks,
Pete
-----Original Message-----
From: Ramon Hofer [mailto:[email protected]]
Sent: Friday, October 14, 2016 9:30 AM
To: Peter Sangas; [email protected]
Subject: Re: "bitmap" line in /proc/mdstat
Quoted 1 line · Peter Sangas
Can you quantify the performance decrease and if rebuild time is not as important as performance how does one remove the bitmap?
From the Linux RAID wiki:
"Some limited measurements I have done suggest up to 10% slowdown, though usually less. Possibly some tuning can make it much better." [1]
[1]
https://raid.wiki.kernel.org/index.php/Write-intent_bitmap#How_Bitmaps_Work_by_Q.26A
Regards,
Ramon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Peter Sangas
Do you know why some of your mds were created with bitmaps and others were not? When I installed Ubuntu 16.04 and used the installer to create a RAID1 some of the mds had bitmaps and some did not.
Yes, I do. Phil Turmel answer my initial question:
Quoted 2 lines · Phil Turmel
Bitmaps have been available for a long time, but including them on a new array by default is recent.
http://marc.info/?l=linux-raid&m=147644598622369&w=2
You can remove the bitmap while the RAID is running with
# mdadm -G /dev/md5 -b none
or add it with
# mdadm -G /dev/md5 -b internal
Regards,
Ramon
Quoted 1 line · Peter Sangas
I installed Ubuntu 16.04 and used the installer to create a RAID1 some of the mds had bitmaps and some did not.
Small ones (/boot, swap) might not get a bitmap by default. According to the manpage it needs 100G or larger.
| -b, --bitmap= |[...] | When creating an array on devices which are 100G or larger, | mdadm automatically adds an internal bitmap as it will usually | be beneficial. This can be suppressed with --bitmap=none .
Regards
Andreas Klauer