Bitmap did not survive reboot
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Notice that bring up all raids normally happens before mount filesystems. So, with your bitmaps on a partition that likely isn't mounted when the raids are brought up, how would it ever work?
It's possible it never has, since as I say I don't ordinarily reboot
these systems. How can I work around this? As you can see from myoriginal
Quoted 1 line
post, the boot and root file systems are reiserfs, and we are cautioned
Quoted 8 lines
not
Quoted 1 line
to put the bitmap on anything other than an ext2 or ext3 file system,
which
Quoted 1 line
is why I created the small
/dev/hd4partition.Sure, that makes sense, but given that usually filesystems are on raid devices and not the other way around, it makes sense that the raids are brought up first ;-)
I see your point. It's certainly a simpler approach.
Quoted 11 lines
Quoted 2 lines
I suppose I could put a script in
/etc/init.det al in order to grow
the array with a bitmap every time the system boots, but that's clumsy,and
Quoted 5 lines
I could imagine it could create a problem if the array ever comes up unclean.
I guess I could also drop to nunlevel 0, wipe the root partition,
recreate it as an ext3, and copy all the files back, but I don't relishthe
Quoted 1 line
idea. Is there any way to convert a reiserfs partition to an ext3 on
the
Quoted 8 lines
Quoted 3 lines
fly?
Oh, and just BTW, Debian does not employ rc.sysinit.
I don't grok debian unfortunately :-( However, short of essentially going in an hand editing the startup files to either A) mount the bitmap mount early or B) both start and mount md0 late, you won't get it to work.
Well, it's supposed to be pretty simple, but I just ran across
something very odd. Instead of using an rc.sysinit file, Debian maintains a directory in /etc for each runlevel named rcN.d, where N is the runlevel, plus one named rcS.d and a file named rc.local. The rc.local is run after exiting any multi-user runlevel, and normally does nothing but quit with an exit code of 0. Generally, the files in the rcN.d and rcS.d directories are all just symlinks to scripts in /etc/init.d. The convention is the link names are of the form Sxxyyyy or Kxxyyyy, where xx is a number between 01 and 99 and yyyy is just some mnemonic text. Any link with a leading "K" is taken to be disabled and is thus ignored by the system
The scripts in rcS.d are executed during a system boot, before
entering any runlevel, including a single user runlevel. In addition to running everything in rcS.d at boot time, whenever entering runlevel N, all the files in rcN.d are executed. Each file is executed in order by its name. Thus, all the S01 - S10 scripts are run before S20, etc. By the time any S40xxx script runs in rcS.d, all the local file systems should be mounted, networking should be available, and all device drivers should be initialized. By the time any S60xxx script is run, the system clock should be set, any NFS file systems should be mounted (unless they depend upon the automounter), and any file system cleaning should be done. The first RAID script in rcS.d is S25mdadm-raid and the first call to the mount script is S35mountall.sh. Thus, as you say, the RAID systems are loaded before the system attempts to mount anything other than /. The default runlevel in Debian is 2, so during ordinary booting, everything in rcS.d should run followed by everything in rc2.d.
Here's what's weird, and it can't really be correct... I don't
think. In both rcS.d and rc2.d (and no doubt others), there are two scripts:
RAID-Server:/etc# ll rcS.d/*md* lrwxrwxrwx 1 root root 20 2008-11-21 22:35 rcS.d/S25mdadm-raid -> ../init.d/mdadm-raid lrwxrwxrwx 1 root root 20 2008-12-27 18:35 rcS.d/S99mdadm_monitor -> ../init.d/mdadm-raid RAID-Server:/etc# ll rc2.d/*md* lrwxrwxrwx 1 root root 15 2008-11-21 22:35 rc2.d/S25mdadm -> ../init.d/mdadm lrwxrwxrwx 1 root root 20 2008-12-27 18:36 rc2.d/S99mdadm_monitor -> ../init.d/mdadm-raid
Note both S99mdadm_monitor links point to /etc/init.d/mdadm-raid,
and so does the S25mdadm-raid script in rc2.d, while the /etc/rc2.d/S25mdadm script points to /etc/init.d/mdadm. The mdadm-raid script starts up the RAID process, and the mdadm script runs the monitor. It seems to me the only link which is really correct is the rcS.d/S25mdadm. At the very least I would think both the S99mdadm_monitor links should point to init.d/mdadm (which, after all is the script which starts the monitor) and that rc2.d/S25mdadm-raid would point to init.d/mdadm, just as the rcS.d/S25mdadm-raid link does. Of course, since the RAID startup script does get called before any of the others, and since the script only shuts down RAID for runlevel 0 (halt) or runlevel 6 (reboot) and not for runlevel 1 - 5 or S, it still works OK, but I don't think it's really correct. Can someone else comment?
Getting back to my dilema, however, I suppose I could simply create
an /etc/rcS.d/S24mounthda4 script that explicitly mounts /dev/hda4 to /etc/mdadm/bitmap, or I could modify the init.d/mdadm-raid script to explicitly mount the /dev/hda partition if it is not already mounted. Editing the init.d/mdadm-raid script is a bit cleaner and perhaps clearer, but any update to mdadm is liable to wipe out the modifications to the startup script.
Quoted 4 lines
If the array isn't super performance critical, I would use mdadm
to delete the bitmap, then grow an internal bitmap with a nice high chunk size and just go from there. It can't be worse than what you've got going on now.
I really dislike that option. Doing it manually every time I boot
would be a pain. Writing a script to do it automatically is no more trouble (or really much different) than writing a script to mount the partition explicitly prior to running mdadm, but it avoids any issues of which I am unaware (but can imagine) with, say, trying to grow a bitmap on an array that is other than clean. I'd rather have mdadm take care of such details.
What do you (and the other memebers of the list) think?
Quoted 8 lines
The mdadm-raid script starts up the
RAID process, and the mdadm script runs the monitor. It seems to me the only link which is really correct is the rcS.d/S25mdadm. At the very least
I would think both theS99mdadm_monitorlinks should point to init.d/mdadm (which, after all is the script which starts the monitor) and that rc2.d/S25mdadm-raid would point to init.d/mdadm, just as the rcS.d/S25mdadm-raid link does.
'Sorry, I meant to say, "rc2.d/S25mdadm-raid would point to
init.d/mdadm-raid, just as the rcS.d/S25mdadm-raid link does." The naming convention here is definitely confusing.
Hello Leslie,
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
Quoted 18 lines · Leslie Rhorer
Quoted 8 lines
The mdadm-raid script starts up the
RAID process, and the mdadm script runs the monitor. It seems to me the only link which is really correct is the rcS.d/S25mdadm. At the very least
I would think both theS99mdadm_monitorlinks should point to init.d/mdadm (which, after all is the script which starts the monitor) and that rc2.d/S25mdadm-raid would point to init.d/mdadm, just as the rcS.d/S25mdadm-raid link does.'Sorry, I meant to say, "rc2.d/S25mdadm-raid would point to init.d/mdadm-raid, just as the rcS.d/S25mdadm-raid link does." The naming convention here is definitely confusing.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Majed B. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the
For 8 Terabytes of data? No, I don't. I'm also not really keen on
interrupting the system ( in whatever fashion ) for six to eight days while I copy the data out and back or taking the primary copy offline while I re-do the array just so I can implement an internal bitmap. It's much easier to handle the external situation one way or the other.
Quoted 2 lines
patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
That's why I am leaning strongly toward the lower value script,
which in fact I have already done. Of course, it's also trivial to disable it. Updating mdadm or the distro won't affect the mount script. At most I would only have to rename the link, and then only if the mdadm startup link gets re-numbered, which is unlilkely. Creating the following script and one symlink to it are hardly "patchwork" in any significant sense:
#! /bin/sh
# Explicitly mount /dev/hda4 prior to running mdadm so the write-intent
# bitmap will be available to mdadm
echo Mounting RAID bitmap...
mount -t ext2 /dev/hda4 /etc/mdadm/bitmap
That kind of exception is one of the main areas I think major dists fail. Making it so absolutely difficult to insert administratively know requirements at 'odd' points in the boot order. When I last used Debian it was easy with that S## / K## linking system. Arch is another dist that has a way of doing that, except it's based in a core config file. I like Debian's method more because then you can use shell scripts to easily slice/dice/add things at given points. Arch is more than sufficient for normal tasks though.
Quoted 4 lines
That kind of exception is one of the main areas I think major dists fail. Making it so absolutely difficult to insert administratively know requirements at 'odd' points in the boot order. When I last used Debian it was easy with that S## / K## linking system.
I agree. It may not be the fastest or the most "sexy" method, but
it is solid, simple, and easy to manage, the oddness I discovered this evening notwithstanding. (The backup server doesn't have the issue, only the main video server did. I simply erased the "duplicate" links, so now RAID is started at boot and the monitor is started on entry to runlevel 2 - 5.)
Quoted 5 lines
Arch is
another dist that has a way of doing that, except it's based in a core config file. I like Debian's method more because then you can use shell scripts to easily slice/dice/add things at given points. Arch is more than sufficient for normal tasks though.
The main reason I like Debian is it stays well away from the
bleeding edge. Most distros have a stable and an unstable version, and some have a testing version. Debian has experimental, testing, unstable, and stable, and its testing version is more like what most distros call their stable version. I do really like the approach Debian takes to its booting. It's really easy to troubleshoot a booting issue. Making changes to the boot sequence often doesn't even require editing any files. One can simply rename a link to a higher or lower number to move it about in the boot sequence, or rename it from Sxxyyy to Kxxyyy to disable it.
It took me less than 3 minutes total to implement the explicit mount
routine on both servers, and unless someone can give me either a much better solution or a solid reason I should not take this approach, I think I'm going to stay with it until such time as I either re-format the root partition or else re-format the array on either respective system. I don't expect the former on either system any time soon. I expect to do the latter on one of the arrays some time in the next three or four months, and the other within a year or so, at which point I may choose the internal bitmap. I don't know, though. I think I rather prefer the external bitmap.
On 11/11/2009 08:53, Leslie Rhorer wrote:
[...]
Quoted 3 lines
It took me less than 3 minutes total to implement the explicit mount
routine on both servers, and unless someone can give me either a much better solution or a solid reason I should not take this approach
The only problem I can see is that your bitmap is not on a RAID device so if the disc goes you lose the bitmap. I guess you're accepting the risk of downtime anyway because your filesystem root is on a non-RAID device, but while you can (as I think you've said before) replace the boot disc quickly, you're exposing yourself to a long, slow resync which will increase your downtime by perhaps days...
Cheers,
John.
Quoted 9 lines · Leslie Rhorer
[...]
Quoted 2 lines
It took me less than 3 minutes total to implement the explicit mount
routine on both servers, and unless someone can give me either a muchbetter
Quoted 1 line
solution or a solid reason I should not take this approach
The only problem I can see is that your bitmap is not on a RAID device so if the disc goes you lose the bitmap. I guess you're accepting the
True, but since the OS is also on the drive, if it goes I will need
to shut down the system anyway.
Quoted 4 lines
risk of downtime anyway because your filesystem root is on a non-RAID device, but while you can (as I think you've said before) replace the boot disc quickly, you're exposing yourself to a long, slow resync which will increase your downtime by perhaps days...
That presumes a loss of the boot drive *AND* one or two RAID drives.
That's pretty unlikely, unless something really nasty happens. The boot drives are not on the same controller or in the same enclosure. They aren't even the same type of drive. The boot drives are PATA drives inside the same enclosure as the respective motherboards. The RAID drives are SATA drives in external RAID enclosures. Right now, a non-bitmap resync takes about a day and a half, if I limit array access.
Quoted 1 line · Leslie Rhorer
Quoted 1 line · Leslie Rhorer
[...]
Quoted 7 lines
Quoted 4 lines
risk of downtime anyway because your filesystem root is on a non-RAID device, but while you can (as I think you've said before) replace the boot disc quickly, you're exposing yourself to a long, slow resync which will increase your downtime by perhaps days...
That presumes a loss of the boot drive *AND* one or two RAID drives.
That's pretty unlikely, unless something really nasty happens.
I don't think it does - even if none of the RAID drives goes, the system crashing because of the root drive going AWOL might well leave the system with the RAID array marked "dirty" because it wasn't shut down cleanly, which without the bitmap would mean you'd get a full resync when you rebooted.
Cheers,
John.
Quoted 1 line
mount -t ext2
/dev/hda4/etc/mdadm/bitmap
I would suggest you mount the partition using UUID, just in case one day the disk decided to change its name, like what happened to me a while back.
Quoted 33 lines · Leslie Rhorer
Quoted 2 lines
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the
For 8 Terabytes of data? No, I don't. I'm also not really keen on interrupting the system ( in whatever fashion ) for six to eight days while I copy the data out and back or taking the primary copy offline while I re-do the array just so I can implement an internal bitmap. It's much easier to handle the external situation one way or the other.
Quoted 2 lines
patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
That's why I am leaning strongly toward the lower value script, which in fact I have already done. Of course, it's also trivial to disable it. Updating mdadm or the distro won't affect the mount script. At most I would only have to rename the link, and then only if the mdadm startup link gets re-numbered, which is unlilkely. Creating the following script and one symlink to it are hardly "patchwork" in any significant sense:
#!
/bin/sh
# Explicitly mount/dev/hda4prior to running mdadm so the write-intent# bitmap will be available to mdadmecho Mounting RAID bitmap...
mount -t ext2/dev/hda4/etc/mdadm/bitmap--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Majed B. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines
Quoted 1 line
mount -t ext2
/dev/hda4/etc/mdadm/bitmapI would suggest you mount the partition using UUID, just in case one day the disk decided to change its name, like what happened to me a while back.
Yeah, I've had that happen with SATA drives quite a bit. That's not
a bad idea.
Quoted 7 lines · Majed B.
Hello Leslie,
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
There should be no need to move the data off - you can add an internal bitmap using the --grow option. An internal bitmap does have more of an overhead than an external one though.
Cheers,
Robin
Signature · 4 lines
___
( ' } | Robin Hill <[email protected]> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |Quoted 9 lines
Quoted 6 lines
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
There should be no need to move the data off - you can add an internal bitmap using the --grow option. An internal bitmap does have more of an overhead than an external one though.
I thought I remembered reading in the man page than an internal
bitmap could only be added when the array was created? Is that incorrect?
Quoted 6 lines · Leslie Rhorer
Quoted 6 lines
Quoted 6 lines
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
Quoted 7 lines
Quoted 3 lines
There should be no need to move the data off - you can add an internal bitmap using the --grow option. An internal bitmap does have more of an overhead than an external one though.
I thought I remembered reading in the man page than an internal
bitmap could only be added when the array was created? Is that incorrect?
I've certainly done this with mdadm 2.6.8 - I guess older versions may not be able to though. It's only able to use a limited amount of space though (whatever's left between the metadata and the data/end of disk), so you don't get as much (if any) control of the chunk size.
Cheers,
Robin
Signature · 4 lines
___
( ' } | Robin Hill <[email protected]> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |Quoted 6 lines · Leslie Rhorer
Quoted 6 lines
Quoted 6 lines
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
Quoted 6 lines
Quoted 3 lines
There should be no need to move the data off - you can add an internal bitmap using the --grow option. An internal bitmap does have more of an overhead than an external one though.
I thought I remembered reading in the man page than an internal
bitmap could only be added when the array was created? Is that incorrect?
Yes, very incorrect. You can use grow to add an internal bitmap later, the only limitation is that the bitmap must be small enough to fit in the reserved space around the superblock. It's in the case that you want to create some super huge, absolutely insanely fine grained bitmap that it must be done at raid device creation time and that's only so it can reserve sufficient space for the bitmap.
Signature · 6 lines
Doug Ledford <[email protected]> GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband
Hi,
Quoted 1 line · Doug Ledford
Yes, very incorrect. You can use grow to add an internal bitmap later,
Is that true for RAID-10, as well? I understood "--grow" with RAID-10 wasn't fully capable -- yet.
BenDJ
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Ben DJ
Hi,
Quoted 1 line · Doug Ledford
Yes, very incorrect. You can use grow to add an internal bitmap later,
Is that true for RAID-10, as well? I understood "--grow" with RAID-10 wasn't fully capable -- yet.
It's true for RAID-10, yes. You can't physically grow the array, but you can definitely add/remove the bitmap.
Cheers,
Robin
Signature · 4 lines
___
( ' } | Robin Hill <[email protected]> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |Quoted 7 lines · Ben DJ
Hi,
Quoted 1 line · Doug Ledford
Yes, very incorrect. You can use grow to add an internal bitmap later,
Is that true for RAID-10, as well? I understood "--grow" with RAID-10 wasn't fully capable -- yet.
I don't know. I never heard anything about raid-10 and grow not being compatible. I'd just set up a couple fakes devices using loopback, create a raid-10, and then try it ;-)
Signature · 6 lines
Doug Ledford <[email protected]> GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband
Quoted 5 lines
Quoted 1 line
would be a pain. Writing a script to do it automatically is no more
trouble
Quoted 2 lines
(or really much different) than writing a script to mount the partition explicitly prior to running mdadm, but it avoids any issues of which I
am
Quoted 8 lines
Quoted 2 lines
unaware (but can imagine) with, say, trying to grow a bitmap on an array that is other than clean. I'd rather have mdadm take care of such
details.
I think you are overestimating the difficulty of this solution. It's as simple as:mdadm -G
/dev/md0--bitmap=none
mdadm -G/dev/md0--bitmap=internal --bitmap-chunk=32768 (or even higher)
No, I was referring to a script which grew an external bitmap on a
mounted file system after mdadm had already done its magic. What I was mis-remembering was:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
If you have a temporary space for your data, I'd suggest you move it out and go for an internal bitmap solution. It certainly beats the patch work you're going to have to do on the startup scripts (and every time you update mdadm, or the distro).
There should be no need to move the data off - you can add an internal bitmap using the --grow option. An internal bitmap does have more of
Quoted 8 lines
an
Quoted 4 lines
Quoted 1 line
overhead than an external one though.
I thought I remembered reading in the man page than an internal
bitmap could only be added when the array was created? Is thatincorrect?
Yes, very incorrect. You can use grow to add an internal bitmap later,
I guess I skimmed over the manual rather quickly back then, and I
was dealing with serious RAID issues at the time, so I must have improperly inferred the man page to imply this in the section which says, "Note that if you add a bitmap stored in a file which is in a filesystem that is on the raid array being affected, the system will deadlock. The bitmap must be on a separate filesystem" to read something more like, "Note that if you add a bitmap ... the bitmap must be on a separate filesystem.
Quoted 5 lines
the only limitation is that the bitmap must be small enough to fit in the reserved space around the superblock. It's in the case that you want to create some super huge, absolutely insanely fine grained bitmap that it must be done at raid device creation time and that's only so it can reserve sufficient space for the bitmap.
How can I know how much space is available? I tried adding the
internal bitmap without specifying anything, and it seems to have worked fine. When I created the bitmap in an external file (without specifying the size), it was around 100K, which seems rather small. Both of these systems use un-partitioned disks with XFS mounted directly on the RAID array. One is a 7 drive RAID5 array on 1.5 TB disks and the other is a 10 drive RAID6 array on 1.0TB disks. Both are using a version 1.2 superblock. The only thing which jumps out at me is --examine, but it doesn't seem to tell me much:
RAID-Server:/usr/share/pyTivo# mdadm --examine /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
Name : RAID-Server:0 (local to host RAID-Server)
Creation Time : Sat Apr 25 01:17:12 2009
Raid Level : raid6
Raid Devices : 10Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : d40c9255:cef0739f:966d448d:e549ada8
Internal Bitmap : 2 sectors from superblock
Update Time : Wed Nov 11 18:17:26 2009
Checksum : 9a4cc480 - correct
Events : 488380
Chunk Size : 256K
Array Slot : 0 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)Array State : Uuuuuuuuuu
Backup:/etc/gadmin-rsync# mdadm --examine /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 940ae4e4:04057ffc:5e92d2fb:63e3efb7
Name : 'Backup':0
Creation Time : Sun Jul 12 20:44:02 2009
Raid Level : raid5
Raid Devices : 7Avail Dev Size : 2930276896 (1397.26 GiB 1500.30 GB)
Array Size : 17581661184 (8383.59 GiB 9001.81 GB)
Used Dev Size : 2930276864 (1397.26 GiB 1500.30 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6156794f:00807e1b:306ed20d:b81914de
Internal Bitmap : 2 sectors from superblock
Update Time : Wed Nov 11 11:52:43 2009
Checksum : 12afc60a - correct
Events : 10100
Layout : left-symmetric
Chunk Size : 256K
Array Slot : 0 (0, 1, 2, 3, 4, 5, 6)
Array State : Uuuuuuu
Quoted 7 lines · Leslie Rhorer
I guess I skimmed over the manual rather quickly back then, and I
was dealing with serious RAID issues at the time, so I must have improperly inferred the man page to imply this in the section which says, "Note that if you add a bitmap stored in a file which is in a filesystem that is on the raid array being affected, the system will deadlock. The bitmap must be on a separate filesystem" to read something more like, "Note that if you add a bitmap ... the bitmap must be on a separate filesystem.
Understandable, and now corrected, so no biggie ;-)
Quoted 10 lines
Quoted 5 lines
the only limitation is that the bitmap must be small enough to fit in the reserved space around the superblock. It's in the case that you want to create some super huge, absolutely insanely fine grained bitmap that it must be done at raid device creation time and that's only so it can reserve sufficient space for the bitmap.
How can I know how much space is available? I tried adding the
internal bitmap without specifying anything, and it seems to have worked fine. When I created the bitmap in an external file (without specifying the size), it was around 100K, which seems rather small.
100k is a huge bitmap. For my 2.5TB array, and a bitmap chunk size of 32768KB, I get the entire in-memory bitmap in 24k (as I recall, the in-memory bitmap is larger than the on-disk bitmap as the on-disk bitmap only stores a dirty/clean bit per chunk where as the in-memory bitmap also includes a counter per chunk so it knows when all outstanding writes complete and it needs to transition to clean, but I could be mis-remembering that).
Quoted 18 lines
Both of these systems
use un-partitioned disks with XFS mounted directly on the RAID array. One is a 7 drive RAID5 array on 1.5 TB disks and the other is a 10 drive RAID6 array on 1.0TB disks. Both are using a version 1.2 superblock. The only thing which jumps out at me is --examine, but it doesn't seem to tell me much:RAID-Server:/usr/share/pyTivo# mdadm --examine /dev/sda /dev/sda: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : RAID-Server:0 (local to host RAID-Server) Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10
Quoted 5 lines
Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
Array Size : 15628197888 (7452.11 GiB 8001.64 GB)Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
The above two items are what you need for both version 1.1 and 1.2 superblocks in order to figure things out. The data, aka the filesystem itself, starts at the Data Offset which is 272 sectors. The superblock itself is 8 sectors in from the front of the disk because you have version 1.2 superblocks. So, 272 - 8 - size of the superblock, which is only a sector or two, is how much internal space you have. So, in your case, you have about 132k of space for the bitmap. Version 1.0 superblocks are a little different in that you need to know the actual size of the device and you need the super offset and possibly the used dev size. There will be free space between the end of the data and the superblock (super offset - used dev size) and free space after the superblock (actual dev size as given by fdisk (either the size of the device itself on whole disk devices or the size of the partition you are using) - super offset - size of superblock). I don't know which is used by the bitmap, but I seem to recall the bitmap wants to be between the superblock and the end of the data, so I think the used dev size and super offset are the important numbers there.
You mentioned that you used the defaults when creating the bitmap. That's likely to hurt your performance. The default bitmap chunk is too small. I would redo it with a larger bitmap chunk. If you look in /proc/mdstat, it should tell you the current bitmap chunk. Given that you stream large sequential files, you could go with an insanely large bitmap chunk and be fine. Something like 65536 or 131072 should be good.
Signature · 6 lines
Doug Ledford <[email protected]> GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband
Quoted 10 lines
Quoted 2 lines
Data Offset : 272 sectors
Super Offset : 8 sectorsThe above two items are what you need for both version 1.1 and 1.2 superblocks in order to figure things out. The data, aka the filesystem itself, starts at the Data Offset which is 272 sectors. The superblock itself is 8 sectors in from the front of the disk because you have version 1.2 superblocks. So, 272 - 8 - size of the superblock, which is only a sector or two, is how much internal space you have. So, in your case, you have about 132k of space for the bitmap.
OK. The 10 drive system shows:
bitmap: 0/466 pages [0KB], 1024KB chunk
The 7 drive system shows:
bitmap: 0/350 pages [0KB], 2048KB chunk
So you think I should remove both and replace them with
mdadm -G /dev/md0 --bitmap=internal --bitmap-chunk=65536
?
While most of the files are large video files, there are a fair
number which are smaller data files such as those of the IMAP server and Quicken. I don't want performance to be too terrible for them, either.
Quoted 21 lines · Leslie Rhorer
Quoted 10 lines
Quoted 2 lines
Data Offset : 272 sectors
Super Offset : 8 sectorsThe above two items are what you need for both version 1.1 and 1.2 superblocks in order to figure things out. The data, aka the filesystem itself, starts at the Data Offset which is 272 sectors. The superblock itself is 8 sectors in from the front of the disk because you have version 1.2 superblocks. So, 272 - 8 - size of the superblock, which is only a sector or two, is how much internal space you have. So, in your case, you have about 132k of space for the bitmap.
OK. The 10 drive system shows:
bitmap: 0/466 pages [0KB], 1024KB chunk
The 7 drive system shows:
bitmap: 0/350 pages [0KB], 2048KB chunk
So you think I should remove both and replace them with
Quoted 7 lines
mdadm -G
/dev/md0--bitmap=internal --bitmap-chunk=65536?
While most of the files are large video files, there are a fair
number which are smaller data files such as those of the IMAP server and Quicken. I don't want performance to be too terrible for them, either.
Oh yeah, those chunk sizes are waaayyyy too small. Definitely replace them. If it will make you feel better, you can do some performance testing before and after to see why I say so ;-) I would recommend running these tests to check the performance change for yourself:
dbench -t 300 -D $mpoint --clients-per-process=4 16 | tail -19 >> $log_file mkdir $mpoint/bonnie
chown nobody.nobody $mpoint/bonnie
bonnie++ -u nobody:nobody -d $mpoint/bonnie -f -m RAID${lvl}-${num}Disk-${chunk}k -n 64:65536:1024:16 >>$log_file 2>/dev/null tiotest -f 1024 -t 6 -r 1000 -d $mpoint -b 4096 >> $log_file tiotest -f 1024 -t 6 -r 1000 -d $mpoint -b 16384 >> $log_file
Obviously, I pulled these tests out of a script I use where all these various variables are defined. Just replace the variables with something sensible for accessing your array, run them, save off the results, run again with a different chunk size, then please post the results back here as I imagine they would be very informative. Especially the dbench results as I think they are likely to benefit the most from the change. Note: dbench, bonnie++, and tiotest should all be available in the debian repos.
Signature · 6 lines
Doug Ledford <[email protected]> GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband
Quoted 10 lines
dbench -t 300 -D $mpoint --clients-per-process=4 16 | tail -19 >> $log_file
mkdir $mpoint/bonnie
chown nobody.nobody $mpoint/bonnie
bonnie++ -u nobody:nobody -d $mpoint/bonnie -f -m RAID${lvl}-${num}Disk-${chunk}k -n 64:65536:1024:16 >>$log_file 2>/dev/null
tiotest -f 1024 -t 6 -r 1000 -d $mpoint -b 4096 >> $log_file tiotest -f 1024 -t 6 -r 1000 -d $mpoint -b 16384 >> $log_file
Quoted 8 lines
Obviously, I pulled these tests out of a script I use where all these various variables are defined. Just replace the variables with something sensible for accessing your array, run them, save off the results, run again with a different chunk size, then please post the results back here as I imagine they would be very informative. Especially the dbench results as I think they are likely to benefit the most from the change. Note: dbench, bonnie++, and tiotest should all be available in the debian repos.
I could not find tiotest. Also, the version dbench in the distro
does not support the --clients-per-process switch. I'll post the results from the backup system here, and from the primary system in the next post.
Backup with bitmap-chunk 65M:
16 363285 109.22 MB/sec execute 286 sec 16 364067 109.05 MB/sec execute 287 sec 16 365960 109.26 MB/sec execute 288 sec 16 366880 109.13 MB/sec execute 289 sec 16 368850 109.35 MB/sec execute 290 sec 16 370444 109.45 MB/sec execute 291 sec 16 372360 109.64 MB/sec execute 292 sec 16 373973 109.74 MB/sec execute 293 sec 16 374821 109.61 MB/sec execute 294 sec 16 376967 109.88 MB/sec execute 295 sec 16 377813 109.77 MB/sec execute 296 sec 16 379422 109.87 MB/sec execute 297 sec 16 381197 110.05 MB/sec execute 298 sec 16 382029 109.92 MB/sec execute 299 sec 16 383868 110.10 MB/sec cleanup 300 sec 16 383868 109.74 MB/sec cleanup 301 sec 16 383868 109.37 MB/sec cleanup 302 sec
Throughput 110.101 MB/sec 16 procs
Version 1.03d ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
RAID5-7Disk-6 3520M 66779 23 46588 22 127821 29 334.7
2
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read----Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
64:65536:1024/16 35 1 113 2 404 9 36 1 54 1 193
5
Backup with default (2048K) bitmap-chunk
16 306080 91.68 MB/sec execute 286 sec 16 307214 91.69 MB/sec execute 287 sec 16 307922 91.61 MB/sec execute 288 sec 16 308828 91.53 MB/sec execute 289 sec 16 310653 91.78 MB/sec execute 290 sec 16 311926 91.82 MB/sec execute 291 sec 16 313569 92.01 MB/sec execute 292 sec 16 314478 91.96 MB/sec execute 293 sec 16 315578 91.99 MB/sec execute 294 sec 16 317416 92.18 MB/sec execute 295 sec 16 318576 92.25 MB/sec execute 296 sec 16 320391 92.39 MB/sec execute 297 sec 16 321309 92.40 MB/sec execute 298 sec 16 322461 92.42 MB/sec execute 299 sec 16 324486 92.70 MB/sec cleanup 300 sec 16 324486 92.39 MB/sec cleanup 301 sec 16 324486 92.17 MB/sec cleanup 302 sec
Throughput 92.6969 MB/sec 16 procs
Version 1.03d ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
RAID5-7Disk-2 3520M 38751 14 31738 15 114481 28 279.0
1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read----Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
64:65536:1024/16 30 1 104 2 340 8 30 1 64 1 160
4
Quoted 34 lines · Leslie Rhorer
Quoted 10 lines
dbench -t 300 -D $mpoint --clients-per-process=4 16 | tail -19 >> $log_file
mkdir $mpoint/bonnie
chown nobody.nobody $mpoint/bonnie
bonnie++ -u nobody:nobody -d $mpoint/bonnie -f -m RAID${lvl}-${num}Disk-${chunk}k -n 64:65536:1024:16 >>$log_file 2>/dev/null
tiotest -f 1024 -t 6 -r 1000 -d $mpoint -b 4096 >> $log_file tiotest -f 1024 -t 6 -r 1000 -d $mpoint -b 16384 >> $log_fileQuoted 8 lines
Obviously, I pulled these tests out of a script I use where all these various variables are defined. Just replace the variables with something sensible for accessing your array, run them, save off the results, run again with a different chunk size, then please post the results back here as I imagine they would be very informative. Especially the dbench results as I think they are likely to benefit the most from the change. Note: dbench, bonnie++, and tiotest should all be available in the debian repos.
I could not find tiotest. Also, the version dbench in the distro
does not support the --clients-per-process switch. I'll post the results from the backup system here, and from the primary system in the next post.Backup with bitmap-chunk 65M:
16 363285 109.22 MB/sec execute 286 sec 16 364067 109.05 MB/sec execute 287 sec 16 365960 109.26 MB/sec execute 288 sec 16 366880 109.13 MB/sec execute 289 sec 16 368850 109.35 MB/sec execute 290 sec 16 370444 109.45 MB/sec execute 291 sec 16 372360 109.64 MB/sec execute 292 sec 16 373973 109.74 MB/sec execute 293 sec 16 374821 109.61 MB/sec execute 294 sec
Quoted 8 lines
16 376967 109.88 MB/sec execute 295 sec 16 377813 109.77 MB/sec execute 296 sec 16 379422 109.87 MB/sec execute 297 sec 16 381197 110.05 MB/sec execute 298 sec 16 382029 109.92 MB/sec execute 299 sec 16 383868 110.10 MB/sec cleanup 300 sec 16 383868 109.74 MB/sec cleanup 301 sec 16 383868 109.37 MB/sec cleanup 302 sec
Hmmm...interesting. This is not the output I expected. This is the second by second update from the app, not the final results. The tail -19 should have grabbed the final results and looked something like this:
Operation Count AvgLat MaxLat
----------------------------------------
NTCreateX 3712699 0.186 297.432 Close 2726300 0.013 168.654 Rename 157340 0.149 161.108 Unlink 750442 0.317 274.044 Qpathinfo 3367128 0.054 297.590 Qfileinfo 586968 0.011 148.788 Qfsinfo 617376 0.921 373.536 Sfileinfo 302636 0.028 151.030 Find 1301556 0.121 309.603 WriteX 1834128 0.125 341.075 ReadX 5825192 0.047 239.368 LockX 12088 0.006 24.543 UnlockX 12088 0.006 23.540 Flush 260391 7.149 520.703
Throughput 385.585 MB/sec 64 clients 16 procs max_latency=661.232 ms
This allows comparison of not just the final throughput but also the various activities. Regardless though, 109 average versus 92 average is a very telling story. That's an 18% performance difference and amounts to a *HUGE* factor.
Quoted 9 lines
Throughput 110.101 MB/sec 16 procs
Version 1.03d ------Sequential Output------ --Sequential Input-
--Random--Per Chr- --Block-- -Rewrite- -Per Chr- --Block----Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
RAID5-7Disk-6 3520M 66779 23 46588 22 127821 29 334.7
2
Quoted 8 lines
------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read----Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
64:65536:1024/16 35 1 113 2 404 9 36 1 54 1 193
5
Ditto with these bonnie++ numbers, *HUGE* difference. 66MB/s versus 38MB/s, 46MB/s versus 31MB/s, and 127MB/s versus 114MB/s on the sequential stuff. The random numbers are all low enough that I'm not sure I trust them (the random numbers in my test setup are in the thousands, not the hundreds).
Quoted 39 lines
Backup with default (2048K) bitmap-chunk
16 306080 91.68 MB/sec execute 286 sec 16 307214 91.69 MB/sec execute 287 sec 16 307922 91.61 MB/sec execute 288 sec 16 308828 91.53 MB/sec execute 289 sec 16 310653 91.78 MB/sec execute 290 sec 16 311926 91.82 MB/sec execute 291 sec 16 313569 92.01 MB/sec execute 292 sec 16 314478 91.96 MB/sec execute 293 sec 16 315578 91.99 MB/sec execute 294 sec 16 317416 92.18 MB/sec execute 295 sec 16 318576 92.25 MB/sec execute 296 sec 16 320391 92.39 MB/sec execute 297 sec 16 321309 92.40 MB/sec execute 298 sec 16 322461 92.42 MB/sec execute 299 sec 16 324486 92.70 MB/sec cleanup 300 sec 16 324486 92.39 MB/sec cleanup 301 sec 16 324486 92.17 MB/sec cleanup 302 secThroughput 92.6969 MB/sec 16 procs
Version 1.03d ------Sequential Output------ --Sequential Input-
--Random--Per Chr- --Block-- -Rewrite- -Per Chr- --Block----Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
RAID5-7Disk-2 3520M 38751 14 31738 15 114481 28 279.0
1------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read----Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
64:65536:1024/16 30 1 104 2 340 8 30 1 64 1 160
4
Signature · 6 lines
Doug Ledford <[email protected]> GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband
Quoted 4 lines
This allows comparison of not just the final throughput but also the various activities. Regardless though, 109 average versus 92 average is a very telling story. That's an 18% performance difference and amounts to a *HUGE* factor.
Well, not so much. Remember, there is only 1 link for ingress /
egress on these machines - a single Gig-E link. Getting much over 90 MBps would be a challenge. Really the only process running on this machine is an rsync daemon which runs at 04:00 every morning, and I really don't care if the rsync takes an extra 10 minutes or some such. Of course, in the event of having to copy the entire data set to a failed array, any extra performance would be welcome, but I'm really not concerned about it. Now if this were one of my commercial production servers, it would be a different matter, but this is for my house, and it is only a backup unit. That doesn't mean I am going to revert to the smaller bitmap chunk, though.
Quoted 9 lines · Leslie Rhorer
RAID-Server:/etc# ll rcS.d/*md* lrwxrwxrwx 1 root root 20 2008-11-21 22:35 rcS.d/S25mdadm-raid -> ../init.d/mdadm-raid lrwxrwxrwx 1 root root 20 2008-12-27 18:35 rcS.d/S99mdadm_monitor -> ../init.d/mdadm-raid RAID-Server:/etc# ll rc2.d/*md* lrwxrwxrwx 1 root root 15 2008-11-21 22:35 rc2.d/S25mdadm -> ../init.d/mdadm lrwxrwxrwx 1 root root 20 2008-12-27 18:36 rc2.d/S99mdadm_monitor -> ../init.d/mdadm-raid
What Debian version do you have? There are no mdadm_monitor links in lenny, and I do not have etch systems anymore to check.
Quoted 7 lines
Getting back to my dilema, however, I suppose I could simply create
an/etc/rcS.d/S24mounthda4script that explicitly mounts/dev/hda4to/etc/mdadm/bitmap, or I could modify the init.d/mdadm-raid script to explicitly mount the/dev/hdapartition if it is not already mounted. Editing the init.d/mdadm-raid script is a bit cleaner and perhaps clearer, but any update to mdadm is liable to wipe out the modifications to the startup script.
In Debian, modifications to init scripts are preserved during upgrade unless you explicitely request them to be overwritten.
Gabor
Signature · 4 lines
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------Quoted 7 lines
Quoted 7 lines
RAID-Server:/etc# ll rcS.d/*md* lrwxrwxrwx 1 root root 20 2008-11-21 22:35 rcS.d/S25mdadm-raid -> ../init.d/mdadm-raid lrwxrwxrwx 1 root root 20 2008-12-27 18:35 rcS.d/S99mdadm_monitor -> ../init.d/mdadm-raid RAID-Server:/etc# ll rc2.d/*md* lrwxrwxrwx 1 root root 15 2008-11-21 22:35 rc2.d/S25mdadm ->
Quoted 6 lines
../init.d/mdadm
Quoted 2 lines
lrwxrwxrwx 1 root root 20 2008-12-27 18:36 rc2.d/S99mdadm_monitor -> ../init.d/mdadm-raid
What Debian version do you have? There are no
mdadm_monitorlinks in lenny, and I do not have etch systems anymore to check.
Lenny. I'm not sure where they came from. It might even have been
me playing around at some point, and I just forgot to delete them. I did a lot of fiddling with mdadm about a year ago.
Quoted 11 lines
Quoted 5 lines
Getting back to my dilema, however, I suppose I could simply create
an/etc/rcS.d/S24mounthda4script that explicitly mounts/dev/hda4to/etc/mdadm/bitmap, or I could modify the init.d/mdadm-raid script to explicitly mount the/dev/hdapartition if it is not already mounted. Editing the init.d/mdadm-raid script is a bit cleaner and perhapsclearer,
Quoted 2 lines
but any update to mdadm is liable to wipe out the modifications to the startup script.
In Debian, modifications to init scripts are preserved during upgrade unless you explicitely request them to be overwritten.
I didn't know that, but even so, it's probably at least somewhat
better not to modify a script unless it's necessary, because the new package might have some important differences in the script.
Quoted 11 lines · Leslie Rhorer
Quoted 4 lines
If the array isn't super performance critical, I would use mdadm
to delete the bitmap, then grow an internal bitmap with a nice high chunk size and just go from there. It can't be worse than what you've got going on now.I really dislike that option. Doing it manually every time I boot
would be a pain. Writing a script to do it automatically is no more trouble (or really much different) than writing a script to mount the partition explicitly prior to running mdadm, but it avoids any issues of which I am unaware (but can imagine) with, say, trying to grow a bitmap on an array that is other than clean. I'd rather have mdadm take care of such details.
I think you are overestimating the difficulty of this solution. It's as simple as:
mdadm -G /dev/md0 --bitmap=none
mdadm -G /dev/md0 --bitmap=internal --bitmap-chunk=32768 (or even higher)
and you are done. It won't need to resync the entire device as the device is already clean and it won't create a bitmap that's too large for the free space that currently exists between the superblock and the start of your data. You can see the free space available for a bitmap by running mdadm -E on one of the block devices and interpreting the data start/data offset/superblock offset fields (sorry there isn't a simply field to look at, but the math changes depending on what superblock version you use and I can't remember if I've ever known what superblock you happen to have). No need to copy stuff around, no need to take things down, all done in place, and the issue is solved permanently with no need to muck around in your system init scripts as from now on when you boot up the bitmap is internal to the array and will be used from the second the array is assembled. The only reason I mentioned anything about performance is because an internal bitmap does slightly slow down random access to an array (although not so much streaming access), but that slowdown is mitigated by using a nice high bitmap chunk size (and for most people a big bitmap chunk is preferable anyway). As I recall, you are serving video files, so your access pattern is large streaming I/O, and that means the bitmap really shouldn't be noticeable in your performance.
Signature · 6 lines
Doug Ledford <[email protected]> GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband