Skip to contentSkip to conversation

Bitmap in RAM?

DISCUSSION Dark Penguin 3 participants 4 messages Oct 8, 2016 – Oct 28, 2016 MD RAID

lore
Share & tools
lore
#0 lore raw

After researching write-intent bitmaps for a while, my understanding is that they are used only to speed up "re-adding" drives by avoiding a full resync, and to enable --write-mostly --write-behind. However, it does introduce some pretty heavy load on whatever device it's on, especially if it's an internal bitmap, because the head would have to fly all the way to the superblock twice per each write. If it's an external bitmap, then the device it's on would be too busy just serving it to do anything else.

So if I were to place it on a tmpfs, I could eliminate this problem only at the expense of being unable to re-add drives after a reboot, right?.. I've read somewhere that bitmaps only work correctly on ext2 or ext3 filesystems, but that probably means that it's not a good idea to put it on a filesystem with delayed allocation like ext4 of zfs, otherwise I don't understand why - and so I don't know if there would be any problem with it running on tmpfs. Is there?..

By the way, Phil, you are a hero! :) I remember that it was you who taught me about the "timeouts and scrubbing" problem a year ago, and you always explain things so well! You must have a lot of patience and love for all people! :)

Signature · 1 line
darkpenguin
#1 lore raw
Quoted 2 lines · On Sat, 8 Oct 2016 15:54:26 +0300 Dark Penguin <[email protected]> wrote:

So if I were to place it on a tmpfs, I could eliminate this problem only at the expense of being unable to re-add drives after a reboot, right?..

If you don't need that ability, you can just remove bitmap entirely, it's not mandatory. Run:

  mdadm --grow --bitmap=none /dev/mdX

However I'd say being able to re-add drives is very useful, so first consider switching to a higher bitmap granularity,

  mdadm --grow --bitmap=none /dev/mdX
  mdadm --grow --bitmap=internal --bitmap-chunk=131072 /dev/mdX

(or even 262144, 524288) as that will reduce the performance impact of the bitmap.

Signature · 2 lines
With respect,
Roman
#2 lore raw

If I remove it entirely, I won't be able to re-add drives at all. If I move it to a tmpfs, then I can re-add them, but not across reboots - and with no downsides, which I want to confirm. So this would be better than removing it completely.

I've thought about it and switched the external bitmaps to the chunk-size of 65536, which apparently is the default for intermal bitmaps. They've become much smaller, which means the default size selected for them before was indeed much higher. I'll see if I notice any difference the next time I'm moving data around; maybe the load will indeed be negligible.

Quoted 20 lines · Roman Mamedov
Quoted 2 lines · On Sat, 8 Oct 2016 15:54:26 +0300 Dark Penguin <[email protected]> wrote:

So if I were to place it on a tmpfs, I could eliminate this problem only at the expense of being unable to re-add drives after a reboot, right?..

If you don't need that ability, you can just remove bitmap entirely, it's not mandatory. Run:

   mdadm --grow --bitmap=none /dev/mdX

However I'd say being able to re-add drives is very useful, so first consider switching to a higher bitmap granularity,

   mdadm --grow --bitmap=none /dev/mdX
   mdadm --grow --bitmap=internal --bitmap-chunk=131072 /dev/mdX

(or even 262144, 524288) as that will reduce the performance impact of the bitmap.

Signature · 1 line
darkpenguin
#3 lore raw
In reply to Dark Penguin
Quoted 3 lines · Dark Penguin

After researching write-intent bitmaps for a while, my understanding is that they are used only to speed up "re-adding" drives by avoiding a full resync, and to enable --write-mostly --write-behind. However, it

This is not correct. Speeding up re-adding is certainly one benefit. The other benefit is speeding up resync after a system crash.

Quoted 5 lines

does introduce some pretty heavy load on whatever device it's on, especially if it's an internal bitmap, because the head would have to fly all the way to the superblock twice per each write. If it's an external bitmap, then the device it's on would be too busy just serving it to do anything else.

It doesn't update the bitmap immediately before and after every write. Writes are batched, and the bitmap is updated once before each batch of writes.
There does need to be another update to record that the write has completed, but that is delayed and usually merged with the update at the start of the next batch of writes.
So the bitmap is usually updated once per batch of writes.

Quoted 8 lines

So if I were to place it on a tmpfs, I could eliminate this problem only at the expense of being unable to re-add drives after a reboot, right?.. I've read somewhere that bitmaps only work correctly on ext2 or ext3 filesystems, but that probably means that it's not a good idea to put it on a filesystem with delayed allocation like ext4 of zfs, otherwise I don't understand why - and so I don't know if there would be any problem with it running on tmpfs. Is there?..

You could create a ramdisk, create an ext2 filesystem on that, and put the bitmap file there.

It probably would make sense to support in-memory bitmaps which never get written to disk. It would be fairly easy to do, and would allow expedited re-add. I just hasn't been done.
(patches welcome :-)

NeilBrown