Skip to contentSkip to conversation

kernel panic on lvcreate

linux-lvm Christopher Hawkins 3 participants 6 messages Dec 9, 2009 – Dec 11, 2009 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Hello,

After some time I revisited this issue on a freshly installed Centos 5.4 box, latest kernel (2.6.18-164.6.1.el5 ) and the panic is still reproducible. Any time I create a snapshot of the root filesystem, kernel panics. The LVM HOWTO says to post bug reports to this list. Is this the proper place?

Thanks,
Chris

From earlier post:
OOPS message:

BUG: scheduling while atomic: java/0x00000001/2959                               [<c061637f>] <3>BUG: scheduling while atomic: java/0x00000001/2867              [<c061637f>] schedule+0x43/0xa55                                                [<c042c40d>] lock_timer_base+0x15/0x2f
 [<c042c46b>] try_to_del_timer_sync+0x44/0x4a
 [<c0437dd2>] futex_wake+0x3c/0xa5
 [<c0434d5f>] prepare_to_wait+0x24/0x46
 [<c0461ea7>] do_wp_page+0x1b3/0x5bb
 [<c0438b01>] do_futex+0x239/0xb5e
 [<c0434c13>] autoremove_wake_function+0x0/0x2d
 [<c0463876>] __handle_mm_fault+0x9a9/0xa15
 [<c041e727>] default_wake_function+0x0/0xc
 [<c046548d>] unmap_region+0xe1/0xf0
 [<c061954f>] do_page_fault+0x233/0x4e1
 [<c061931c>] do_page_fault+0x0/0x4e1
 [<c0405a89>] error_code+0x39/0x40
 =======================
schedule+0x43/0xa55
 [<c042c40d>] <0>------------[ cut here ]------------

kernel BUG at arch/i386/mm/highmem.c:43! invalid opcode: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq

Modules linked in: autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc ip6t_REJECTdCPU:    3 ip6table_filter ip6_tables x_tables ipv6 xfrm_nalgo cry
EIP:    0060:[<c041cb08>]    Not tainted VLI
EFLAGS: 00010206   (2.6.18-164.2.1.el5 #1)

EIP is at kmap_atomic+0x5c/0x7f
eax: c0012d6c ebx: fff5b000 ecx: c1fb8760 edx: 00000180
esi: f7be8580 edi: f7fa7000 ebp: 00000004 esp: f5c54f0c
ds: 007b es: 007b ss: 0068 Process mpath_wait (pid: 3273, ti=f5c54000 task=f5c50000 task.ti=f5c54000)ne Stack: c073a4e0 c0462f7f f7b0eb30 f7b40780 f5c54f3c 0029c3f0 f63b5ef0 f7be8580

       f7b40780 f7fa7000 00008802 c0472d75 f7b0eb30 f7c299c0 00001000 00001000
       00001000 00000101 00000001 00000000 00000000 f5c5007b 0000007b ffffffff
Call Trace:
 [<c0462f7f>] __handle_mm_fault+0xb2/0xa15
 [<c0472d75>] do_filp_open+0x2b/0x31
 [<c061954f>] do_page_fault+0x233/0x4e1
 [<c061931c>] do_page_fault+0x0/0x4e1
 [<c0405a89>] error_code+0x39/0x40
 =======================
Code: 00 89 e0 25 00 f0 ff ff 6b 50 10 1b 8d 14 13 bb 00 f0 ff ff 8d 42 44 c1 e EIP: [<c041cb08>] kmap_atomic+0x5c/0x7f SS:ESP 0068:f5c54f0c
 <0>Kernel panic - not syncing: Fatal exception

0c 29 c3 a1 54 12 79 c0 c1 e2 02 29 d0 83 38 00 74 08 <0f> 0b 2b

Quoted 21 lines · ----- "Milan Broz
Quoted 1 line · Christopher Hawkins

When I create a root snapshot on a fairly typical Centos 5.3

server:
...

Quoted 1 line

I get a kernel panic.

Please try to first update kernel to version from 5.4. (There were some fixes for snapshot like
https://bugzilla.redhat.com/show_bug.cgi?id=496100)

If it still fails, please post the OOps trace from kernel (syslog).

Milan
--
[email protected]

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
Quoted 3 lines · Christopher Hawkins

After some time I revisited this issue on a freshly installed Centos 5.4 box, latest kernel (2.6.18-164.6.1.el5 ) and the panic is still reproducible. Any time I create a snapshot of the root filesystem, kernel panics.

I guess it is already reported here https://bugzilla.redhat.com/show_bug.cgi?id=539328
so please watch this bugzilla.

Milan
--
[email protected]

#4 lore raw
In reply to Christopher Hawkins
Quoted 4 lines · Christopher Hawkins

After some time I revisited this issue on a freshly installed Centos 5.4 box, latest kernel (2.6.18-164.6.1.el5 ) and the panic is still reproducible. Any time I create a snapshot of the root filesystem, kernel panics. The LVM HOWTO says to post bug reports to this list. Is this the proper place?

Bummer. I would post the bug on Centos bugzilla also. Please post the bug number here if you do it (cause I'll get to it eventually).

Thanks for testing this. I have the same problem, and have a new client to install by next year - so not much time to work on it.

Now that we know it is not yet fixed, we can form theories as to what is going wrong. My guess is that the problem is caused by the fact that lvm is updating files in /etc/lvm on the root filesystem while taking the snapshot. These updates are done by user space programs, so I would further speculate that *any* snapshot would crash if an update happened exactly when creating the snapshot - i.e. the atomic nature of snapshot creation has been broken. The lvm user space probably does fsync() on files
in /etc/lvm, which might be involved in triggering the crash.

We could test the first theory by moving /etc/lvm to another volume (I sometimes put it on /boot - a non LVM filesystem - for easier disaster recovery.) Naturally, I wouldn't go moving /etc/lvm on a production server.

Testing the second hypothesis is less certain, and would basically involve trying snapshots of LVs undergoing heavy updating.

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#5 lore raw

It is reported here:

https://bugzilla.redhat.com/show_bug.cgi?id=539328

That is definitely the one. And it sounds like they have a potential fix... I have already emailed the developers there asking if I can help test their patch, so hopefully soon I can post back and report status.

Christopher Hawkins

Quoted 56 lines · ----- "Stuart D. Gathman
Quoted 1 line · Christopher Hawkins

After some time I revisited this issue on a freshly installed Centos

5.4 box,

Quoted 1 line

latest kernel (2.6.18-164.6.1.el5 ) and the panic is still

reproducible. Any

Quoted 1 line

time I create a snapshot of the root filesystem, kernel panics. The

LVM HOWTO

Quoted 1 line

says to post bug reports to this list. Is this the proper place?

Bummer. I would post the bug on Centos bugzilla also. Please post the
bug number here if you do it (cause I'll get to it eventually).

Thanks for testing this. I have the same problem, and have a new client
to install by next year - so not much time to work on it.

Now that we know it is not yet fixed, we can form theories as to what is going wrong. My guess is that the problem is caused by the fact that
lvm is updating files in /etc/lvm on the root filesystem while taking the snapshot. These updates are done by user space programs, so I would
further speculate that *any* snapshot would crash if an update happened exactly
when creating the snapshot - i.e. the atomic nature of snapshot creation has
been broken. The lvm user space probably does fsync() on files in /etc/lvm, which might be involved in triggering the crash.

We could test the first theory by moving /etc/lvm to another volume (I
sometimes put it on /boot - a non LVM filesystem - for easier disaster
recovery.) Naturally, I wouldn't go moving /etc/lvm on a production server.

Testing the second hypothesis is less certain, and would basically involve
trying snapshots of LVs undergoing heavy updating.

Signature · 13 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703
591-6154
"Confutatis maledictis, flammis acribus addictis" - background song
for
a Microsoft sponsored "Where do you want to go from here?"
commercial.

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
#3 lore raw
In reply to Milan Broz
Quoted 4 lines · Milan Broz
Quoted 1 line

and the panic is still reproducible. Any time I create a snapshot of the root filesystem, kernel panics.

I guess it is already reported here https://bugzilla.redhat.com/show_bug.cgi?id=539328
so please watch this bugzilla.

Ah, they are getting it while Oracle is updating. That supports the updates during supposedly atomic snapshot theory.

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#6 lore raw
In reply to Christopher Hawkins

There is a working patch for this issue in kernel-2.6.18-178.el5. If anyone needs it sooner, they do have a test kernel available for download.

Christopher Hawkins

Quoted 79 lines · ----- "Christopher Hawkins

It is reported here:

https://bugzilla.redhat.com/show_bug.cgi?id=539328

That is definitely the one. And it sounds like they have a potential fix... I have already emailed the developers there asking if I can help test their patch, so hopefully soon I can post back and report status.

Christopher Hawkins

Quoted 3 lines · ----- "Stuart D. Gathman
Quoted 1 line · Christopher Hawkins

After some time I revisited this issue on a freshly installed

Centos

Quoted 4 lines

5.4 box,

Quoted 1 line

latest kernel (2.6.18-164.6.1.el5 ) and the panic is still

reproducible. Any

Quoted 1 line

time I create a snapshot of the root filesystem, kernel panics.

The

Quoted 7 lines

LVM HOWTO

Quoted 1 line

says to post bug reports to this list. Is this the proper place?

Bummer. I would post the bug on Centos bugzilla also. Please post the
bug number here if you do it (cause I'll get to it eventually).

Quoted 5 lines

Thanks for testing this. I have the same problem, and have a new client
to install by next year - so not much time to work on it.

Now that we know it is not yet fixed, we can form theories as to

what

Quoted 3 lines

is going wrong. My guess is that the problem is caused by the fact that
lvm is updating files in /etc/lvm on the root filesystem while

taking

Quoted 16 lines

the snapshot. These updates are done by user space programs, so I would
further speculate that *any* snapshot would crash if an update happened exactly
when creating the snapshot - i.e. the atomic nature of snapshot creation has
been broken. The lvm user space probably does fsync() on files in /etc/lvm, which might be involved in triggering the crash.

We could test the first theory by moving /etc/lvm to another volume (I
sometimes put it on /boot - a non LVM filesystem - for easier disaster
recovery.) Naturally, I wouldn't go moving /etc/lvm on a production server.

Quoted 18 lines

Testing the second hypothesis is less certain, and would basically involve
trying snapshots of LVs undergoing heavy updating.

Signature · 13 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703
591-6154
"Confutatis maledictis, flammis acribus addictis" - background song
for
a Microsoft sponsored "Where do you want to go from here?"
commercial.

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/