md: can not impport hdb1, has active inodes!

Debian stable 2.4.19 kernel with RAID1 compiled in, VIA EPIA CL1000 motherboard, 2 Maxtor DiamondMax 9 7200rpm 80Gb drives as /dev/hdb and /dev/hdc from onboard controller.

I've followed the guide at:
http://www.cs.montana.edu/faq/faqw.admin.py?query=Convert+Root+System+
to+Software+Raid&querytype=simple&casefold=yes&req=search

which had turned up on a debian list search to try to set up the RAID1 mirror of the boot/root drive. I installed to /dev/hdb1 (/dev/hda is the CDROM and perhaps I should have changed that first).

Loaded 2.2.20, got 2.4.19 sources, compiled RAID1 support into the kernel (still can't get rid of one complaint about a missing character set module but don't think that's causing any real problems), made the identical /dev/hdc1 into a somewhat smaller linux

autodetect RAID format partition, mounted it as 2nd drive in a RAID1 drive using:
mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/hdc1
which worked fine, mkfs -t ext3 /dev/hdc1, mount it to /mnt1, cp -ax / /mnt, tweak /etc/fstab ... /dev/md0 mounts fine, tweak lilo conf to boot from /dev/md0, reboot -- fine, finally come to add /dev/hdb1 into the array after resetting the partition type to autodetect RAID ...

.. no go, system complains that /dev/hdb1 is mounted or that an inode

is active:
md: can not impport hdb1, has active inodes!
md: error, md_import_device() returned -16

I assume that's because the boot is still from /dev/hdb but if so, how come the howto says this should work. Seems that nothing I do can shift this and if I set the partition back to linux I find I can't mount the drive despite rebooting as I get:

mount: /dev/hdb1 already mounted or /mnt1 busy

/mnt1 isn't busy (I can create a completely new mount point: no difference) and mount shows that /dev/hdb1 isn't mounted.

I've tried all sorts of tweaking with lilo.conf to remove any calls to /dev/hdb but either these don't change anything, or else lilo complains that it can't install to the target (e.g. /dev/md0).

I've read through the docs and man pages and done some searching around but can't find any explanations.

Help of any sort, including direct pointers to what I've missed or misread, desperately needed!

Chris

PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

I've been battling the same problem using Debian testing/sarge with SCSI devices. I followed (mostly) the same howto and have subsequently tried many other things with no success. I'm willing to help debug this as well if anyone has any advice. Thanks

Cameron Moore
/ The other day, I went to a tourist information booth and asked, \
\  "Tell me about some of the people who were here last year".    /

Aha, that's interesting, so it's true for Sarge too. Thanks Cameron.
I do think there's a real problem here and that it needs fixing and
documenting ... and I'm sure someone out there has worked out the fix... _please_!

C
PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

Wrong.
mkfs -t ext3 /dev/md0
then mount /dev/md0.
Once you have included /dev/hdc1 in an array, don't touch it again - just access the array (/dev/md0).

You missed a step (step 9). You have to reboot so that /dev/md0 is your root device.
Once you have done that and are happy with it, you add in /dev/hdb1 and let it resync.

NeilBrown

Sorry, I was being hasty in writing the Email, that's what I did, i.e. mkfs the /dev/md0 drive. That's not the problem.

Ditto: did that, no go.

Thanks,

C
PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

Sorry, I missed where you said that you rebooted.

It sounds very odd.

What does "cat /proc/mounts" say?

Does your kernel use an initrd?

Exactly how did you tell lilo to use /dev/md0?
With
root=/dev/md0
as an option in lilo.conf, or with
append= "root=/dev/md0"

or both or something else?

NeilBrown

you are correct: I didn't say that, but I have done!

Well, I thought so too, but I see I'm not alone in experiencing this.

Aha, didn't know that wrinkle: lovely. Mount says:
/dev/md0 on / type ext3 (rw,errors=remount-ro) proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)

and cat /proc/mounts says:
rootfs / rootfs rw 0 0
/dev/root / ext2 rw 0 0
proc /proc proc rw 0 0
devpts /dev/pts devpts rw 0 0

Does that "/dev/root" bit indicate anything useful?

No, at least, I haven't asked it to!

I think I've only tried the former (with and without an earlier line, before the image stanzas, saying root=/dev/hdb1)

I haven't used an "append" clause anywhere.

One problem is keeping in mind which of the lilo.conf tweaks installed with lilo but didn't solve the problem, and which were the ones that lilo refused to install.

I'm happy to try alternatives and report effects, on or off list, of each.

Many thanks,

Chris

PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

No, but the fact that root is actually 'ext2' even though mount (and presumably fstab) thinks it is ext3 is a bit suspicious.

If there are any "root=/dev/hdb1" lines in lilo.conf, I suggest taking them out.
I think you currently have /dev/hdb1 as your root filesystem still. You can easily check this:

mkdir /a /b
mount /dev/hdb1 /a
mount /dev/md0 /b

ls -l /?/a-file

If the ls finds "/a/a-file", then /dev/hdb1 is root. If it finds /b/a-file, then /dev/md0 is root.

If it still doesn't work, post your lilo.conf

NeilBrown

I'm splitting my response to this post into two. I'm sure that someone out there who knows about mount and /proc/mounts can clarify this bit:

I suppose that might be because the difference between ext2 and ext3, as I understand it, is just a journalling block reserved at the end of the normal ext2 space so perhaps /proc/mounts indicates all ext3 as ext2?

From a quick bit of googling, I understand that /dev/root is a label for the root mount point that's passed to the kernel as it loads, in my case, I assume that's passed to the kernel from whatever lilo has done to the mbr and that this ensures that the kernel then holds that mount open (presumably in case it needs to write something back there when it quits ... or perhaps it always writes something back there in a clean dismount ... I don't know).

Can someone:
a) confirm or deny my hunch that the ext2/ext3 issue is normal?
b) clarify the way that lilo and the kernel interact?

I wonder if it's pertinent that /proc/partitions shows this:

cat /proc/partitions

major minor  #blocks  name
   9     0   78124928 md0
  22     0   80043264 hdc
  22     1   78125008 hdc1
   3    64   80043264 hdb
   3    65   80035798 hdb1

cfdisk /dev/hdb shows a single primary bootable partition occupying the entire drive of Linux raid autodetect type. For /dev/hdc I have a single primary bootable partition of the same type and some free space (to ensure that there'd be enough room on /dev/hdb1 to add it to the array).

Thanks,

Chris
PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

A reasonable hunch, but no, it is not normal for an ext3 root fs to show up as ext2 in /proc/mounts:

# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
[snip]

Perhaps you don't have ext3 support built in to the kernel? If not, you'll need an initrd to boot your root fs as ext3.

As far as the active inodes on /dev/hdb1, one thing you might try is:

fuser -vm /dev/hdb1

If there's a process accessing /dev/hdb1, it will be listed.

--
Paul

No, it is normal. It happens, when /etc/fstab still contains ext2 as fs-type for the root-fs. I don't recall the exact reason, but it has something to do with the fact that the mount command normally updates /etc/mtab when a filesystem is mounted. As the root-fs is first mounted read-only, mount cannot update /etc/mtab. When the root-fs gets remounted read-write, mount updates /etc/mtab, but (I think) has at that stage no clue of what filesystem type the root-fs is and therefore uses the entry from /etc/fstab. For an authorative answer search the mailing list archive from the ext3 mailing list.

Regards,
Juri

Juri Haberland  <[email protected]> 

Hope it's not too obvious, but...
Did you also actually _run_ lilo ?
Is fstab on /dev/md0 edited to reflect the new mountpoints ?

Not that it helps you here, but I followed the procedure -albeit not from the same howto[1]- multiple times with many machines. It _should_ work...

[1] I used a howto by the name of Boot+root+raid+llilo. Maybe see if there are any obvious differences between the two procedures / howtos...?

Maarten

No it's not too obvious and maybe I need to work through all the options again and document what went wrong each time but I did try to run lilo on each: some ran without complaint but didn't solve the problem, and some resulted in lilo complaining something along the lines that it was using the current boot device and that device x (a numeric representatiion, sorry, didn't note it) wasn't a device in which it could write its stuff.

Well, it encourages me!

Yes, found that. Mostly the differences are that the one I worked from uses mdadm and that one uses raidtools2 but I thought they said the same things really and I couldn't get either approach: mdadm or raidtools/raidtab, to work for me.

One thing: I asssume that mdrecoveryd isn't holding an inode active? I _think_ I've failed just as totally after killing mdrecoveryd but can't vouch for having done that every time.

Thanks Maarten,

C

PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

Neil had some interesting points, but in the meantime here's my working lilo.conf (which was written by the SuSE installer) in case it helps you:

boot = /dev/hda
change-rules
reset
default = linux
disk=/dev/hda

    bios=0x80
    disk=/dev/hdc
    bios=0x81

lba32
prompt
read-only
timeout = 80

image = /boot/vmlinuz
label = linux
initrd = /boot/initrd
root = /dev/md0

Greetings,
Maarten

I thought I'd got this all cracked in the early hours of this morning. I _think_ I'd been putting boot=/dev/hda1 when I tried to write the revised lilo config to the MBR and perhaps that messed up hdb. I don't know, anyway, I started again, did everything in that howto pretty much to the letter and this time, after shifting boot to /dev/hdc and rebooting, I could add /dev/hdb1. However, lilo would not let me use a raid-extra-boot line and when I rebooted, the system hung right at the start.

I moved hdb1 to hda1 and started again, this time it's all worked except for boot=/dev/md0 and the raid-extra-boot. Now lilo allows these and says it's written to both MBRs, but when I reboot lilo gets to the first twenty or so of the dots after "Loading Linux" and hangs there ... resolutely.

A lot of pratting around with rescuing from the installation CD (must work out how to burn a rescue boot CDROM) and I am back to being able to get everything find if I boot from boot=/dev/hdc with root=/dev/md0 ... and I think I'll give up here as I must go back to doing my real job and having a life. If anyone can tell me what might be neeeded (I haven't tried things like bios=0x80 lines) I may find the energy to try them but not if this machine has become my working firewall!

So I'm left thinking there's something, perhaps idiosyncratic to some BIOS or HDs or whatever, that means some of us have real problems writing LILO boot instructions to the two drives in a RAID1 boot array.

I've leflt Maarten's listing of a SuSE lilo.conf to see if anyone has ideas.

Thank everyone,

Chris

PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

Good questions Justin and I'm increasingly realising that I needed an almost forensic attitude to this and to take it much more slowly and tap into much more information that was there on the machine. However, I have no reason to believe that the kernel wasn't the right one but next time I will make sure I check carefully.

I am increasingly tempted to do this and there is room there (there's 936Mb as it happens: I think the first hard disc I looked after for anyone was 10Mb and it was the sole hard drive in that XT box!)

My sense is that this would be simpler and I'd feel less scared of it but I'd be slightly less robust as there'd be no reserve boot point if that failed (though if it did I should be able to rescue from a
boot floppy or CD I guess). I think that's not that different from
continuing to have "boot=/dev/hda" (or "boot=/dev/hdc") in lilo.conf, as you note you have, I think that too is givign you a single MBR boot record.

I am still being slow about this though. One thing that's very clear now is that if I let lilo (22.2) write its boot to the two drive's mbrs with the raid-extra-boot, something goes badly wrong. As I really need to finish this saga, even if I still don't really understand what was wrong, I am going to stick with my current lilo setting which seems to work which says:
boot=/dev/hda

However, if I do want to use that bit of spare drive to give myself the reassuring feeling that the lilo/mbr issues are being kept away from the /dev/md0 areas, then is this the right sequence:

a) cfdisk to create bootable, linux type, partition /dev/hdc2
b) reboot
c) mkfs -t ext3 /dev/hdc2
d) rewrite lilo.conf to instruct it to boot from /dev/hdc2, if so, does that mean simply writing "boot=/dev/hdc2"? Surely not as I think that means I need some primary boot loader to come out of one of the drive mbrs that will then point to the lilo secondary load from /dev/hdc2 (sorry, I'm sure this is dumb of me but someone take pity here please!)
e) assuming that works and reboots OK
f) init 1
g) stop /dev/md0 (after umount of / ?) can this be done
h) cfdisk /dev/hda and take bootable off, ditto /dev/hdc
i) reboot and pray
j) ... ugh, no this all sounds wrong

I am continuing to ask (and thanks again to Justin, Chris, Neil and Maarten for their inputs) as I do want to feel I understand this and get as safe a set up as I can for this machine, but also because I will have to return to the issues in the next month or two, hopefully while it's still fresh in my mind, to put in a replacement server behind this firewall.

Hence one more question: I had been planning to put at least three drives in that in a RAID5 array and boot/root from that, now I'm rattled but would still like that redundancy. How difficult is RAID5 boot/root cf RAID1?

TIA,

Chris

P.S. I promise to document all of this as some sort of mini-HOWTO or whatever to complement the existing ones, and to notify the authors of those where I think they might usefully be improved: clearly I owe the open source movement at least that much.
PSYCTC: Psychotherapy, Psychology, Psychiatry, Counselling
and Therapeutic Communities; practice, research, teaching and consultancy.
Chris Evans & Jo-anne Carlyle
http://psyctc.org/ Email: [email protected]

Hi Chris.

Just a few points and musings... may be helpful, may be not...

Yeah I forgot to remind you to change hda [my setup] to hdb [yours].

I may have a partial solution for you... I tend to have BIG problems booting off anything but the first BIOS drive, maybe that's lilo, maybe that me not fully understanding lilo (or maybe a BIOS issue even). In any case, this leads to me _always_ having hda as bootdisk. So, when I built my raid0 sets I think I might have done it thusly:

(1) Install your OS on hda
Define raidtab: hda=failed-disk hdc=raid-disk; activate raid, format, cp data Mount /dev/md0 /mnt
Change ONLY the fstab and lilo.conf on the mounted /dev/md0 volume:
tell fstab to find / as /dev/md0
tell lilo.conf to boot off /dev/hda (!hdA! not hdc nor md0)
Now when I would run 'lilo -r /mnt' it will complain "not on the first disk" or something like that... So instead, I do:
Down system; swap over hda with hdc; boot (keep a bootfloppy around!!) run lilo (which must / should work now without complaining)
add the (now free) hdc (which used to be hda) to your raid volume.
(3) Profit ! ;-)

I know, its a hassle, but maybe this does help you a bit though.

Another possibility, but which requires changing your partitions:

After some problems I decided it was simpler to just work around lilo problems by having a separate tiny non-raid /boot partition[*]. This has its drawbacks but it solves the lilo issues. The downside is that you must manually mirror /dev/hda1 to /dev/hdc1 [typically using dd], (also after a kernel upgrade!) and that when disk hda fails it will NOT boot from hdc by itself (it requires an admin with a boot/rescuefloppy physically at the console). If (big IF) you can live with that you may want to try it but be warned that it is somewhat of an ugly hack...

[*] Note: I did this partly for raid but also partly for reiserfs. I thought it prudent to have a plain old ext2 /boot partition with the full kernel as opposed to needing an initrd with reiser support on a single reiserfs partition. YMMV...

Weird... I have never seen that. "LI" and "LIL" errors, yeah, but not that.

Hmm. Bugger... I'm no lilo expert (far from it in fact) but my understanding is / was that especially these lines are ESSENTIAL...!
Why did you omit these lines... They are there for a reason.

I hope you do find a solution... good luck!

Maarten