Oops when running snapshots
Ran a load test last night where we created snapshots two snapshots every 2 hours under heavy file system load. We got the following oops:
Unable to handle kernel NULL pointer dereference at virtual address 00000000 802895f5 *pde = 55bd7001 Oops: 0002 CPU: 2 EIP: 0010:[<802895f5>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010206
eax: 00000000 ebx: f8ab2118 ecx: f8a79048 edx: f8a79030
esi: 00810180 edi: b4db0000 ebp: 00000820 esp: a2765c4c
ds: 0018 es: 0018 ss: 0018
Process vsftpd (pid: 7309, stackpage=a2765000)
Stack: b4db0000 b4db0170 f69b3170 00ea8000 00000080 00000006 00000070
00000000
80286031 a2765caa a2765cac 00810180 b4db0000 00003a02 e338c840
00800070
00ea8000 00000000 00000000 f69b3000 f68a0000 00810180 00000800
08208cbc
Call Trace: [<80286031>] [<80286117>] [<80219ccc>] [<80219d41>] [<801d209f>]
[<801d26eb>] [<801d2ac7>] [<801d2b2a>] [<801387fd>] [<801d2952>]
[<80139b
1d327e>] [<80137807>] [<80106f27>]
Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04
Quoted 7 lines
Quoted 7 lines
EIP;
802895f5<lvm_snapshot_remap_block+a9/f8> <=====ebx;
f8ab2118<[nfstracker].bss.end+490c5/ffe98fa9> ecx;f8a79048<[nfstracker].bss.end+fff5/ffe98fa9> edx;f8a79030<[nfstracker].bss.end+ffdd/ffe98fa9> edi;b4db0000<_end+3499065c/7847565c>
esp;a2765c4c<_end+223462a8/7847565c>
Trace; 80286031 <lvm_map+3b9/490>
Trace; 80286117 <lvm_make_request_fn+f/1c>
Trace; 80219ccc <generic_make_request+11c/12c>
Trace; 80219d41 <submit_bh+65/80>
Trace; 801d209f <submit_page+57/74>
Trace; 801d26eb <page_state_convert+40b/4a0>
Trace; 801d2ac7 <linvfs_writepage+3b/e0>
Trace; 801d2b2a <linvfs_writepage+9e/e0>
Trace; 801387fd <write_some_buffers+9d/14c>
Trace; 801d2952 <linvfs_get_block+1e/24>
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8> <=====
0: 89 10 mov %edx,(%eax) <=====
Code; 802895f7 <lvm_snapshot_remap_block+ab/f8>
2: c7 01 00 00 00 00 movl $0x0,(%ecx)
Code; 802895fd <lvm_snapshot_remap_block+b1/f8>
Trace; 80219d41 <submit_bh+65/80>
Trace; 801d209f <submit_page+57/74>
Trace; 801d26eb <page_state_convert+40b/4a0>
Trace; 801d2ac7 <linvfs_writepage+3b/e0>
Trace; 801d2b2a <linvfs_writepage+9e/e0>
Trace; 801387fd <write_some_buffers+9d/14c>
Trace; 801d2952 <linvfs_get_block+1e/24>
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8> <=====
0: 89 10 mov %edx,(%eax) <=====
Code; 802895f7 <lvm_snapshot_remap_block+ab/f8>
2: c7 01 00 00 00 00 movl $0x0,(%ecx)
Code; 802895fd <lvm_snapshot_remap_block+b1/f8>
8: c7 41 04 00 00 00 00 movl $0x0,0x4(%ecx)
Code; 80289604 <lvm_snapshot_remap_block+b8/f8>
f: 8b 03 mov (%ebx),%eax
Code; 80289606 <lvm_snapshot_remap_block+ba/f8>
11: 89 48 04 mov %ecx,0x4(%eax)
10 warnings issued. Results may not be reliable.
This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of memory and is configured with HIGHME64G. We are running the latest 2.4.21 XFS and are using xfs_freeze to quiesce the file system. We are not using the VFS_LOCK patch.
Anyone have any ideas what may be causing this?
Thanks,
Andrew
Signature · 3 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Andrew Patterson Voice: (970) 898-3261 Hewlett-Packard Company Email: [email protected]
Quoted 7 lines · Andrew Patterson
Ran a load test last night where we created snapshots two snapshots every 2 hours under heavy file system load. We got the following oops:
Unable to handle kernel NULL pointer dereference at virtual address00000000
802895f5
<SNIP>
Quoted 11 lines
11: 89 48 04 mov %ecx,0x4(%eax)
10 warnings issued. Results may not be reliable.
This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of memory and is configured with HIGHME64G. We are running the latest 2.4.21 XFS and are using
xfs_freezeto quiesce the file system. We are not using theVFS_LOCKpatch.Anyone have any ideas what may be causing this?
VM problems ? (we had some reports ITR)
Do you have a chance to retest without HIGHMEM to see if it still fails ?
Quoted 17 lines
Thanks,
Andrew
Signature · 11 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Andrew Patterson Voice: (970) 898-3261 Hewlett-Packard Company Email: [email protected] _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Quoted 19 lines · Heinz J . Mauelshagen
Quoted 7 lines · Andrew Patterson
Ran a load test last night where we created snapshots two snapshots every 2 hours under heavy file system load. We got the following oops:
Unable to handle kernel NULL pointer dereference at virtual address00000000
802895f5<SNIP>
Quoted 10 lines
11: 89 48 04 mov %ecx,0x4(%eax)
10 warnings issued. Results may not be reliable.
This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of memory and is configured with HIGHME64G. We are running the latest 2.4.21 XFS and are using
xfs_freezeto quiesce the file system. We are not using theVFS_LOCKpatch.
Quoted 5 lines
Quoted 1 line
Anyone have any ideas what may be causing this?
VM problems ? (we had some reports ITR)
Do you have a chance to retest without HIGHMEM to see if it still fails ?
I looked through mailing list archives about this. Most the of himem problems resulted in vmmalloc.c calls in the oops. I'll try it again without himem to see if this helps.
Thanks,
Andrew
Quoted 2 lines
Quoted 2 lines
Thanks,
Quoted 19 lines · Heinz J . Mauelshagen
Quoted 7 lines · Andrew Patterson
Ran a load test last night where we created snapshots two snapshots every 2 hours under heavy file system load. We got the following oops:
Unable to handle kernel NULL pointer dereference at virtual address00000000
802895f5<SNIP>
Quoted 10 lines
11: 89 48 04 mov %ecx,0x4(%eax)
10 warnings issued. Results may not be reliable.
This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of memory and is configured with HIGHME64G. We are running the latest 2.4.21 XFS and are using
xfs_freezeto quiesce the file system. We are not using theVFS_LOCKpatch.
Quoted 5 lines
Quoted 1 line
Anyone have any ideas what may be causing this?
VM problems ? (we had some reports ITR)
Do you have a chance to retest without HIGHMEM to see if it still fails ?
I finally had a chance to rerun with HIMEM turned off. I also made one other change. We are now using the linux-2.4.21-VFS-lock.patch I found in the device mapper CVS archives instead of using xfs_freeze (we thought we were running into some race condition with xfs_freeze. Some other details:
We are running multiple snapshots on the same volume. The snapshots and data set changes are very large (constantly changing up to 12 GB using random and sequential read/writes).
We have 2 GB of RAM on the system (the kernel only uses 1 GB due to no HIMEM).
We then get the following oops after creating one or more snapshots:
ksymoops 2.4.8 on i686 2.4.21. Options used
-v /tmp/vmlinux (specified)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.21/ (default)
-m /boot/System.map (specified)
Warning (compare_maps): ksyms_base symbol acpi_fadt_R__ver_acpi_fadt not found in vmlinux. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol acpi_gbl_FADT_R__ver_acpi_gbl_FADT not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_get_best_pio_mode_R__ver_ide_get_best_pio_mode not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_pci_register_driver_R__ver_ide_pci_register_driver not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_pci_unregister_driver_R__ver_ide_pci_unregister_driver not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_pio_timings_R__ver_ide_pio_timings not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_set_xfer_rate_R__ver_ide_set_xfer_rate not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_setup_pci_device_R__ver_ide_setup_pci_device not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_setup_pci_devices_R__ver_ide_setup_pci_devices not found in vmlinux. Ignoring ksyms_base entry
Unable to handle kernel NULL pointer dereference at virtual address 00000000 c0286b55 *pde = 00000000 Oops: 0002 deadman multipath md bonding1 bonding cpqci cpqhealth cpqrom e100 lpfcdd CPU: 0 EIP: 0010:[<c0286b55>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax: 00000000 ebx: f9083e50 ecx: f8acbbf0 edx: f8aa19b0
esi: 0140e500 edi: f3967000 ebp: 00004400 esp: f7e17e58
ds: 0018 es: 0018 ss: 0018
Process kswapd (pid: 11, stackpage=f7e17000)
Stack: f3967000 f3967170 f3e3e770 01d50000 00000080 00000001 00000058 00000000
c0283581 f7e17eb6 f7e17eb8 01400180 f3967000 00003a01 e5187800 013fe3d8
01d50000 00000020 00000200 f3e3e600 f3c4a000 01400180 000013f0 44001000
Call Trace: [<c0283581>] [<c0283667>] [<c021892c>] [<c02189a1>] [<c013a6ac>] [<c013a85f>] [<c0138c70>] [<c012f812>] [<c012fb10>] [<c012fb7c>] [<c012fc81>] [<c012fce6>] [<c012fdfd>] [<c0105684>]
Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04Quoted 7 lines
Quoted 7 lines
EIP;
c0286b55<lvm_snapshot_remap_block+a9/f8> <=====ebx;
f9083e50<[deadman].bss.end+654695/ffed2841>
ecx;f8acbbf0<[deadman].bss.end+9c435/ffed2841>
edx;f8aa19b0<[deadman].bss.end+721f5/ffed2841>
edi;f3967000<_end+33564cdc/38492cdc>
esp;f7e17e58<_end+37a15b34/38492cdc>
Trace; c0283581 <lvm_map+3b9/490>
Trace; c0283667 <lvm_make_request_fn+f/1c>
Trace; c021892c <generic_make_request+120/130>
Trace; c02189a1 <submit_bh+65/80>
Trace; c013a6ac <sync_page_buffers+94/ac>
Trace; c013a85f <try_to_free_buffers+19b/1c4>
Trace; c0138c70 <try_to_release_page+44/48>
Trace; c012f812 <shrink_cache+24e/3cc>
Trace; c012fb10 <shrink_caches+78/a8>
Trace; c012fb7c <try_to_free_pages_zone+3c/5c>
Trace; c012fc81 <kswapd_balance_pgdat+41/8c>
Trace; c012fce6 <kswapd_balance+1a/30>
Trace; c012fdfd <kswapd+99/b4>
Trace; c0105684 <arch_kernel_thread+28/38>
Code; c0286b55 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code; c0286b55 <lvm_snapshot_remap_block+a9/f8> <=====
0: 89 10 mov %edx,(%eax) <=====
Code; c0286b57 <lvm_snapshot_remap_block+ab/f8>
2: c7 01 00 00 00 00 movl $0x0,(%ecx)
Code; c0286b5d <lvm_snapshot_remap_block+b1/f8>
8: c7 41 04 00 00 00 00 movl $0x0,0x4(%ecx)
Code; c0286b64 <lvm_snapshot_remap_block+b8/f8>
f: 8b 03 mov (%ebx),%eax
Code; c0286b66 <lvm_snapshot_remap_block+ba/f8>
11: 89 48 04 mov %ecx,0x4(%eax)
9 warnings issued. Results may not be reliable.
It looks like we are trying to shrink the buffer cache to release more memory for the block remap. This then oopses. Anyone have any idea how to fix this?
Andrew
Signature · 3 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Andrew Patterson Voice: (970) 898-3261 Hewlett-Packard Company Email: [email protected]