hangs when mounting snapshot

Over the past few weeks I have been playing around with LVM with great success. However, I have been experimenting with snapshots recently and run into a problem. I have tried this with reiserfs and ext2 as the filesystem (not that it should matter to LVM but I mention it because mount is involved) and get identical results.

I am using kernel 2.4.0-test6 and lvm_0.8final.

I created a snapshot like so:

# lvcreate --size 16m --snapshot --name snap /dev/test_vg/test_lv 
lvcreate -- WARNING: all snapshots will be disabled if more than 16 MB are changed
lvcreate -- INFO: using default snapshot chunk size of 64 KB
lvcreate -- doing automatic backup of "test_vg"
lvcreate -- logical volume "/dev/test_vg/snap" successfully created

Everything looks cool:

# lvscan
lvscan -- ACTIVE   "/dev/another_test_vg/another_test_lv" [300 MB]
lvscan -- ACTIVE   "/dev/test_vg/test_lv" [63.91 GB] striped[2]
lvscan -- ACTIVE   "/dev/test_vg/snap" [63.91 GB] striped[2]
lvscan -- 3 logical volumes with 128.12 GB total in 2 volume groups
lvscan -- 3 active logical volumes

Then I try to mount it with:

# mount /dev/test_vg/snap /snap1
Segmentation fault

Then I try:

# mount /dev/test_vg/snap /snap1

which hangs the mount command. The rest of the system is ok but not even kill -9 can get rid of that mount command. The system won't reboot properly because it hangs when trying to unmount filesystems and I have to hit the switch.

Any ideas? Do I need to update my version of mount to support snapshots or something?

Tracy Reed      http://www.ultraviolet.org

Yes, I get the same. kdb says:

Entering kdb (0xc1780000) Panic: divide error
due to panic @ 0xc0195f81
eax = 0x000013fc ebx = 0xc1b6e370 ecx = 0xc1b6e200 edx = 0x00000000 esi = 0x000013fe edi = 0x00000002 esp = 0xc1781d44 eip = 0xc0195f81 ebp = 0xc1781d68 ss = 0x00000018 cs = 0x00000010 eflags = 0x00010246
ds = 0xc2aa0018 es = 0x00000018 origeax = 0xffffffff &regs = 0xc1781d10
kdb>

and the backtrace says:

kdb> bt
EBP EIP Function(args)
0xc1781d68 0xc0195f81 lvm_snapshot_remap_block+0x11 (0xc1781de6, 0xc1781de8, 0x1
3fc, 0xc1b6e200)

                               kernel .text 0xc0100000 0xc0195f70 0xc019605c
0xc1781dec 0xc0198281 lvm_map+0x3dd (0xc17e7f80, 0x0)
                               kernel .text 0xc0100000 0xc0197ea4 0xc01982d8
0xc1781dfc 0xc0198314 lvm_make_request_fn+0x10 (0xc0391350, 0x0, 0xc17e7f80)
                               kernel .text 0xc0100000 0xc0198304 0xc0198320
0xc1781e24 0xc018d6d2 generic_make_request+0xb6 (0x0, 0xc17e7f80)
                               kernel .text 0xc0100000 0xc018d61c 0xc018d734
0xc1781e44 0xc018d81a ll_rw_block+0xe6 (0x0, 0x1, 0xc1781e58)
                               kernel .text 0xc0100000 0xc018d734 0xc018d884
0xc1781e5c 0xc012c239 bread+0x31 (0x3a01, 0x1, 0x400)
                               kernel .text 0xc0100000 0xc012c208 0xc012c270
0xc1781ec8 0xc014a96a ext2_read_super+0xb6 (0xc1b6ee00, 0xc16fd000, 0x0)
                               kernel .text 0xc0100000 0xc014a8b4 0xc014af64
0xc1781ee8 0xc012efb5 read_super+0x105 (0x3a01, 0xc1274420, 0xc02ef870, 0x1,
0xc
16fd000)
                               kernel .text 0xc0100000 0xc012eeb0 0xc012f00c
0xc1781f38 0xc012f19c get_sb_bdev+0x128 (0xc02ef870, 0xc16fe000, 0x1,
0xc16fd000
)
                               kernel .text 0xc0100000 0xc012f074 0xc012f1f0
0xc1781f8c 0xc012fcaa do_mount+0x19e (0xc16fe000, 0xc380c000, 0xc16ff000,
0xc0ed
0001, 0xc16fd000)
                               kernel .text 0xc0100000 0xc012fb0c 0xc012fdc0
0xc1781fbc 0xc012fe3c sys_mount+0x7c (0x8058408, 0x8058420, 0x8058438,
0xc0ed000
1, 0x8058468)
                               kernel .text 0xc0100000 0xc012fdc0 0xc012fe80
more>
           0xc0108e23 system_call+0x33
                               kernel .text 0xc0100000 0xc0108df0 0xc0108e28

kdb>

Hope this helps.

D

-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Tracy R Reed
Sent: Thursday, September 21, 2000 6:50 AM
To: [email protected]
Subject: [linux-lvm] hangs when mounting snapshot

Over the past few weeks I have been playing around with LVM with great success.
However, I have been experimenting with snapshots recently and run into a problem. I have tried this with reiserfs and ext2 as the filesystem (not that
it should matter to LVM but I mention it because mount is involved) and get identical results.

I am using kernel 2.4.0-test6 and lvm_0.8final.

I created a snapshot like so:

# lvcreate --size 16m --snapshot --name snap /dev/test_vg/test_lv
lvcreate -- WARNING: all snapshots will be disabled if more than 16 MB are
changed
lvcreate -- INFO: using default snapshot chunk size of 64 KB
lvcreate -- doing automatic backup of "test_vg"
lvcreate -- logical volume "/dev/test_vg/snap" successfully created

Everything looks cool:

# lvscan
lvscan -- ACTIVE   "/dev/another_test_vg/another_test_lv" [300 MB]
lvscan -- ACTIVE   "/dev/test_vg/test_lv" [63.91 GB] striped[2]
lvscan -- ACTIVE   "/dev/test_vg/snap" [63.91 GB] striped[2]
lvscan -- 3 logical volumes with 128.12 GB total in 2 volume groups
lvscan -- 3 active logical volumes

Then I try to mount it with:

# mount /dev/test_vg/snap /snap1
Segmentation fault

Then I try:

# mount /dev/test_vg/snap /snap1

which hangs the mount command. The rest of the system is ok but not even kill
-9 can get rid of that mount command. The system won't reboot properly because
it hangs when trying to unmount filesystems and I have to hit the switch.

Any ideas? Do I need to update my version of mount to support snapshots or something?

--
Tracy Reed http://www.ultraviolet.org

Hello again. I upgraded to util-linux-2.10o which includes the latest version of mount and it is still hanging when mounting snapshots. It segfaults first, then hangs on the next attempt. Is there anything else I need to upgrade to get this working?

Tracy Reed      http://www.ultraviolet.org

The segmentation fault is happening inside the kernel itself, so the problem is not with the mount command, but with LVM. It will have recorded an "OOPS" record in /var/log/messages, and depending if you have syslogd set up correctly, it would tell us where this is happening. From what another poster said, it is happening inside lvm_snapshot_remap_block(), but since I've never really played with snapshots, I can't really comment...

Do you know if you have the latest patches for the user tools, which are not included in the 0.8final tools you download from the LVM site? These fix several longstanding problems in the user tools which may (or may not) be causing your problem. The list of existing patches was posted by me to this mailing list on Aug 4, as "Additions to LVM-BUGS-0.8final", which would be a good starting point. As I don't work with 2.4 lately, I'm not sure if the in-kernel LVM files are up-to-date with all of the fixes... I know the 2.2 support is far behind 2.4, so I keep my own set of kernel patches.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Right, I should have noticed it earlier but it did indeed leave an oops in the messages file. I have run it through ksymoops:

No modules in ksyms, skipping objects
Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid lsmod file?

Sep 25 13:58:57 reiser kernel: CPU:    1 
Sep 25 13:58:57 reiser kernel: EIP:    0010:[lvm_snapshot_remap_block+17/244] 
Sep 25 13:58:57 reiser kernel: EFLAGS: 00010246 
Sep 25 13:58:57 reiser kernel: eax: 000007e0   ebx: dfacc770   ecx: dfacc600
edx: 00000000 
Sep 25 13:58:57 reiser kernel: esi: 00000000   edi: 00000000   ebp: 00000000
esp: df0d5c34 
Sep 25 13:58:57 reiser kernel: ds: 0018   es: 0018   ss: 0018 
Sep 25 13:58:57 reiser kernel: Process mount (pid: 555, stackpage=df0d5000) 
Sep 25 13:58:57 reiser kernel: Stack: dfacc770 00000000 00000000 00000000
00000000 0805734c 0000134c 00000000  
Sep 25 13:58:57 reiser kernel:        c0120018 c01aabb3 df0d5cce df0d5cd0
000007e0 dfacc600 df1ffe40 00000002  
Sep 25 13:58:57 reiser kernel:        000000e8 00000002 c0146a6c 0805734c
e0848000 c0147d5b 0805734c 0805734c  
Sep 25 13:58:57 reiser kernel: Call Trace: [sysctl_string+204/348]
[lvm_map+1095/1208] [padzero+28/32] [<e0848000>] [load_elf_binary+3023/3408]
[load_elf_binary+0/3408] [lvm_make_request_fn+15/24]  
Sep 25 13:58:57 reiser kernel: Code: f7 b1 64 01 00 00 8b 44 24 2c 89 d3 8b 30

89 f7 29 df 89 f8
Using defaults from ksymoops -t elf32-i386 -a i386

Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol
0: f7 b1 64 01 00 00 divl 0x164(%ecx),%eax
Code; 00000006 Before first symbol
6: 8b 44 24 2c movl 0x2c(%esp,1),%eax
Code; 0000000a Before first symbol
a: 89 d3 movl %edx,%ebx
Code; 0000000c Before first symbol
c: 8b 30 movl (%eax),%esi
Code; 0000000e Before first symbol
e: 89 f7 movl %esi,%edi
Code; 00000010 Before first symbol
10: 29 df subl %ebx,%edi
Code; 00000012 Before first symbol
12: 89 f8 movl %edi,%eax

1 warning issued. Results may not be reliable.

No, I do not have those patches. I am having these problems under 2.4 so my problem may or may not be the same as the other posters problem. He was getting the oops in lvm_snapshot_remap_block but it looks like I am getting it in lvm_make_request_fn.

Tracy Reed      http://www.ultraviolet.org

                                               ^^^^^^^^^^^^^^^^^^^^^^^^

[snip]

Actually, it looks like you are having the same problem. From what I can see, the only thing that is used as a divisor is "chunk_size" (inside hash_fn in lvm_find_exception, which is inline).

Is it possible that your user tools are not up-to-date with the kernel LVM headers? It appears that the lv_chunk_size field is after the silly __unused field in the lv_v2_t struct (as are all of the other snapshot fields). This may be throwing things off, because if you look at tools/lib/lvm.h, this field isn't there...

I fixed this in my LVM tools RPM by using only the kernel lvm.h file to ensure they always match. Try adding the "uint8_t __unused" field to lv_v2_t in tools/lib/lvm.h, recompile, and see if this fixes your problem...

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

I went back and looked over the 5 patches in the message you refer to. None of them seem to have anything to do snapshots. I applied them anyway and they do not seem to have helped.

Has anyone had success with snapshots and if so what kernel, user tools, etc?

Tracy Reed      http://www.ultraviolet.org