Corruption of in-memory data detected. Shutting down filesystem

Jul 13 02:24:29 backupserver kernel: [679408.490589] Filesystem "dm-4": XFS internal error xfs_trans_cancel at line 1163 of file fs/xfs/xfs_trans.c. Caller 0xffffffffa0212709
Jul 13 02:24:29 backupserver kernel: [679408.564416] Pid: 18658, comm: rdiff-backup Not tainted 2.6.26-2-amd64 0000001
Jul 13 02:24:29 backupserver kernel: [679408.605393]
Jul 13 02:24:29 backupserver kernel: [679408.605394] Call Trace:
Jul 13 02:24:29 backupserver kernel: [679408.639377] [<ffffffffa0212709>] :xfs:xfs_create+0x3ea/0x477
Jul 13 02:24:29 backupserver kernel: [679408.674393] [<ffffffffa020ca4e>] :xfs:xfs_trans_cancel+0x55/0xed
Jul 13 02:24:29 backupserver kernel: [679408.711940] [<ffffffffa0212709>] :xfs:xfs_create+0x3ea/0x477
Jul 13 02:24:29 backupserver kernel: [679408.758863] [<ffffffffa01d60ca>] :xfs:xfs_attr_get+0x8d/0x9e
Jul 13 02:24:29 backupserver kernel: [679408.794026] [<ffffffffa021b7b7>] :xfs:xfs_vn_mknod+0x159/0x247
Jul 13 02:24:29 backupserver kernel: [679408.830013] [<ffffffff802a2cbd>] vfs_create+0xcf/0x140
Jul 13 02:24:29 backupserver kernel: [679408.862025] [<ffffffff802a587a>] do_filp_open+0x1e3/0x7c4
Jul 13 02:24:29 backupserver kernel: [679408.895520] [<ffffffff802999a2>] do_sys_open+0x46/0xc3
Jul 13 02:24:29 backupserver kernel: [679408.928542] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
Jul 13 02:24:29 backupserver kernel: [679408.967513]
Jul 13 02:24:29 backupserver kernel: [679408.978394] xfs_force_shutdown(dm-4,0x8) called from line 1164 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffa020ca67
Jul 13 02:24:30 backupserver kernel: [679409.051147] Filesystem "dm-4": Corruption of in-memory data detected. Shutting down filesystem: dm-4
Jul 13 02:24:30 backupserver kernel: [679409.106937] Please umount the filesystem, and rectify the problem(s)
Jul 13 02:24:33 backupserver kernel: [679412.440595] Filesystem "dm-4": xfs_log_force: error 5 returned.
Jul 13 02:24:54 backupserver kernel: [679433.470623] Filesystem "dm-4": xfs_log_force: error 5 returned.
[...]
Jul 13 09:53:48 backupserver kernel: [707379.613578] Filesystem "dm-4": xfs_log_force: error 5 returned.

Jul 13 09:53:48 backupserver kernel: [707379.673603] xfs_force_shutdown(dm-4,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffffa0214e2a
Jul 13 09:54:10 backupserver kernel: [707401.740318] Filesystem "dm-4": xfs_log_force: error 5 returned.
[...]
Jul 13 10:03:19 backupserver kernel: [707957.520753] Filesystem "dm-4": xfs_log_force: error 5 returned.
Jul 13 10:03:36 backupserver kernel: [707974.761993] Filesystem "dm-4": Disabling barriers, not supported by the underlying device
Jul 13 10:03:36 backupserver kernel: [707974.857711] XFS mounting filesystem dm-4
Jul 13 10:03:36 backupserver kernel: [707975.126237] Starting XFS recovery on filesystem: dm-4 (logdev: internal)
Jul 13 10:03:50 backupserver kernel: [707991.756505] Ending XFS recovery on filesystem: dm-4 (logdev: internal)

Hi all,

Occasionally I get the above error on one of our backup volumes. The volume is a LVM-volume which resides inside a crypted partition. We have three backup volumes, but only one has the problem. On this volume we backup tons of Subversion working copies, so that mostly very small files are on this volume. The error only appears when our backup via rdiff-backup is running.

I currently work around this problem by unmounting the volume and mounting it again. I would appreciate your help. Let me know, if you need any further information.

Some details

kern.log

Quoted 21 lines

Jul 13 02:24:29 backupserver kernel: [679408.490589] Filesystem "dm-4": XFS internal error xfs_trans_cancel at line 1163 of file fs/xfs/xfs_trans.c. Caller 0xffffffffa0212709
Jul 13 02:24:29 backupserver kernel: [679408.564416] Pid: 18658, comm: rdiff-backup Not tainted 2.6.26-2-amd64 0000001
Jul 13 02:24:29 backupserver kernel: [679408.605393]
Jul 13 02:24:29 backupserver kernel: [679408.605394] Call Trace:
Jul 13 02:24:29 backupserver kernel: [679408.639377] [<ffffffffa0212709>] :xfs:xfs_create+0x3ea/0x477
Jul 13 02:24:29 backupserver kernel: [679408.674393] [<ffffffffa020ca4e>] :xfs:xfs_trans_cancel+0x55/0xed
Jul 13 02:24:29 backupserver kernel: [679408.711940] [<ffffffffa0212709>] :xfs:xfs_create+0x3ea/0x477
Jul 13 02:24:29 backupserver kernel: [679408.758863] [<ffffffffa01d60ca>] :xfs:xfs_attr_get+0x8d/0x9e
Jul 13 02:24:29 backupserver kernel: [679408.794026] [<ffffffffa021b7b7>] :xfs:xfs_vn_mknod+0x159/0x247
Jul 13 02:24:29 backupserver kernel: [679408.830013] [<ffffffff802a2cbd>] vfs_create+0xcf/0x140
Jul 13 02:24:29 backupserver kernel: [679408.862025] [<ffffffff802a587a>] do_filp_open+0x1e3/0x7c4
Jul 13 02:24:29 backupserver kernel: [679408.895520] [<ffffffff802999a2>] do_sys_open+0x46/0xc3
Jul 13 02:24:29 backupserver kernel: [679408.928542] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
Jul 13 02:24:29 backupserver kernel: [679408.967513]
Jul 13 02:24:29 backupserver kernel: [679408.978394] xfs_force_shutdown(dm-4,0x8) called from line 1164 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffa020ca67
Jul 13 02:24:30 backupserver kernel: [679409.051147] Filesystem "dm-4": Corruption of in-memory data detected. Shutting down filesystem: dm-4
Jul 13 02:24:30 backupserver kernel: [679409.106937] Please umount the filesystem, and rectify the problem(s)
Jul 13 02:24:33 backupserver kernel: [679412.440595] Filesystem "dm-4": xfs_log_force: error 5 returned.
Jul 13 02:24:54 backupserver kernel: [679433.470623] Filesystem "dm-4": xfs_log_force: error 5 returned.
[...]
Jul 13 09:53:48 backupserver kernel: [707379.613578] Filesystem "dm-4": xfs_log_force: error 5 returned.

Quoted 8 lines

Jul 13 09:53:48 backupserver kernel: [707379.673603] xfs_force_shutdown(dm-4,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffffa0214e2a
Jul 13 09:54:10 backupserver kernel: [707401.740318] Filesystem "dm-4": xfs_log_force: error 5 returned.
[...]
Jul 13 10:03:19 backupserver kernel: [707957.520753] Filesystem "dm-4": xfs_log_force: error 5 returned.
Jul 13 10:03:36 backupserver kernel: [707974.761993] Filesystem "dm-4": Disabling barriers, not supported by the underlying device
Jul 13 10:03:36 backupserver kernel: [707974.857711] XFS mounting filesystem dm-4
Jul 13 10:03:36 backupserver kernel: [707975.126237] Starting XFS recovery on filesystem: dm-4 (logdev: internal)
Jul 13 10:03:50 backupserver kernel: [707991.756505] Ending XFS recovery on filesystem: dm-4 (logdev: internal)

Kernel-version is 2.6.26-2-amd64 and the OS is a Debian Lenny. The system is up to date and I made sure that the hardware is running fine.

The main causes of this problem were fixed some time ago, but after 2.6.26 was released. IIRC it was in the 2.6.29-30 time frame. Upgrade your kernel....

The main causes of this problem were fixed some time ago, but after 2.6.26 was released. IIRC it was in the 2.6.29-30 time frame. Upgrade your kernel....