vmap allocation for size 1048576 failed
Dear List!
On one of my machines i started getting errors like this after upgrading from Kernel 2.6.35 to 2.6.36.
--- --- snip --- ---
Nov 9 11:25:57 xxxxx kernel: [591880.880679] vmap allocation for size 1048576 failed: use vmalloc=<size> to increase size. ... (repeats a few times, with differing sizes) Nov 9 11:25:57 xxxxx kernel: [591880.881651] xfs_buf_get: failed to map pages ... (repeats a few times) Nov 9 11:25:57 xxxxx kernel: [591881.111625] Filesystem "loop0": XFS internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c. Caller 0xc0270288 Nov 9 11:25:57 xxxxx kernel: [591881.111629] Nov 9 11:25:57 xxxxx kernel: [591881.111636] Pid: 24899, comm: rsync Not tainted 2.6.36 #1 Nov 9 11:25:57 xxxxx kernel: [591881.111640] Call Trace: Nov 9 11:25:57 xxxxx kernel: [591881.111653] [<c0254756>] xfs_error_report+0x2c/0x2e Nov 9 11:25:57 xxxxx kernel: [591881.111663] [<c026c691>] xfs_trans_cancel+0x4b/0xc9 Nov 9 11:25:57 xxxxx kernel: [591881.111674] [<c0270288>] ? xfs_create+0x473/0x4ef Nov 9 11:25:57 xxxxx kernel: [591881.111683] [<c0270288>] xfs_create+0x473/0x4ef Nov 9 11:25:57 xxxxx kernel: [591881.111693] [<c02791c7>] xfs_vn_mknod+0xca/0x156 Nov 9 11:25:57 xxxxx kernel: [591881.111700] [<c027926c>] xfs_vn_create+0xa/0xc Nov 9 11:25:57 xxxxx kernel: [591881.111709] [<c01bcedd>] vfs_create+0x85/0xcf Nov 9 11:25:57 xxxxx kernel: [591881.111716] [<c01bd56e>] do_last+0x277/0x4f8 Nov 9 11:25:57 xxxxx kernel: [591881.111725] [<c01bed11>] do_filp_open+0x197/0x458 Nov 9 11:25:57 xxxxx kernel: [591881.111733] [<c01bd0fb>] ? getname+0x1b/0xb9 Nov 9 11:25:57 xxxxx kernel: [591881.111741] [<c01b3766>] do_sys_open+0x48/0xc9 Nov 9 11:25:57 xxxxx kernel: [591881.111746] [<c01b3829>] sys_open+0x1e/0x26 Nov 9 11:25:57 xxxxx kernel: [591881.111752] [<c0102798>] sysenter_do_call+0x12/0x28 Nov 9 11:25:57 xxxxx kernel: [591881.111759] xfs_force_shutdown(loop0,0x8) called from line 1797 of file fs/xfs/xfs_trans.c. Return address = 0xc026c6a7 Nov 9 11:25:57 xxxxx kernel: [591881.214442] Filesystem "loop0": Corruption of in-memory data detected. Shutting down filesystem: loop0 Nov 9 11:25:57 xxxxx kernel: [591881.214449] Please umount the filesystem, and rectify the problem(s) Nov 9 11:26:11 xxxxx kernel: [591894.827022] Filesystem "loop0": xfs_log_force: error 5 returned. Nov 9 11:26:41 xxxxx kernel: [591924.827024] Filesystem "loop0": xfs_log_force: error 5 returned. Nov 9 11:27:11 xxxxx kernel: [591954.827023] Filesystem "loop0": xfs_log_force: error 5 returned. Nov 9 11:27:18 xxxxx kernel: [591961.412716] xfs_force_shutdown(loop0,0x1) called from line 994 of file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xc0275738 Nov 9 11:27:18 xxxxx kernel: [591961.660660] Filesystem "loop0":
xfs_log_force: error 5 returned.
--- --- snip --- ---
A few thoughts:
* It always happens when i run "rsync" on the machine.
* I've already tried setting vmalloc=256M and vmalloc=512M which both
delayed the problem but didn't cure it.
* It's a 32-bit machine and the problem only occurs on a loop'ed XFS,
never on the root XFS.
* I've used the experimental "delaylog" mount option, which may be
important.
* I'm not sure if the problem is 100% related to the kernel upgrade
because i didn't use a loop XFS on this machine before the kernel upgrade.
Any ideas how to narrow down the problem?
tia,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · On Dienstag, 9. November 2010 Michael Weissenbacher wrote:
* I'm not sure if the problem is 100% related to the kernel upgrade because i didn't use a loop XFS on this machine before the kernel upgrade.
You should describe the loop setup and all devices down to the disk. Might be easier for the devs to help then.
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Radiointerview zum Thema Spam ****** http://www.it-podcast.at/archiv.html#podcast-100716 // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines
You should describe the loop setup and all devices down to the disk. Might be easier for the devs to help then.
-rw-r--r-- 1 root root 2.0G Nov 9 13:01 portage-loopback
/dev/loop0 /mnt/portage xfs rw,noatime,attr2,delaylog,logbsize=256k,noquota
meta-data=/dev/loop0 isize=256 agcount=4, agsize=1048576 blks
= sectsz=512 attr=2
data = bsize=512 blocks=4194304, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=512 blocks=65536, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0/dev/sda3 / xfs rw,swalloc,attr2,delaylog,logbsize=256k,sunit=128,swidth=384,usrquota,grpquota
meta-data=/dev/sda3 isize=256 agcount=4, agsize=90929920
blks
= sectsz=512 attr=2
data = bsize=4096 blocks=363719632, imaxpct=10
= sunit=16 swidth=48 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0/dev/sda is a 4-disk RAID-5 on a 3ware 8506-4LP Controller.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 23 lines · Michael Weissenbacher
Dear List!
On one of my machines i started getting errors like this after upgrading from Kernel 2.6.35 to 2.6.36.
--- --- snip --- ---Nov 9 11:25:57 xxxxx kernel: [591880.880679] vmap allocation for size 1048576 failed: use vmalloc=<size> to increase size. ... (repeats a few times, with differing sizes) Nov 9 11:25:57 xxxxx kernel: [591880.881651] xfs_buf_get: failed to map pages ... (repeats a few times) Nov 9 11:25:57 xxxxx kernel: [591881.111625] Filesystem "loop0": XFS internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c. Caller 0xc0270288 Nov 9 11:25:57 xxxxx kernel: [591881.111629] Nov 9 11:25:57 xxxxx kernel: [591881.111636] Pid: 24899, comm: rsync Not tainted 2.6.36 #1 Nov 9 11:25:57 xxxxx kernel: [591881.111640] Call Trace: Nov 9 11:25:57 xxxxx kernel: [591881.111653] [<c0254756>] xfs_error_report+0x2c/0x2e Nov 9 11:25:57 xxxxx kernel: [591881.111663] [<c026c691>] xfs_trans_cancel+0x4b/0xc9 Nov 9 11:25:57 xxxxx kernel: [591881.111674] [<c0270288>] ?
Quoted 1 line
xfs_create+0x473/0x4ef
I didn't think anything other than log recovery tries to vmap buffers. This is clearly not in log recovery. Can you post an unedited error log, how much data you are rsyncing, the configuration of your filesystem (xfs_info, mount options, loop dev config, etc) to give us an idea of what you are doing to trigger this?
Quoted 6 lines
A few thoughts:
* It always happens when i run "rsync" on the machine.
* I've already tried setting vmalloc=256M and vmalloc=512M which both
delayed the problem but didn't cure it.
* It's a 32-bit machine and the problem only occurs on a loop'ed XFS,
never on the root XFS.
Can't you run on a 64-bit machine?
Quoted 2 lines
* I've used the experimental "delaylog" mount option, which may be
important.
Should be completely irrelevant.
Quoted 2 lines
* I'm not sure if the problem is 100% related to the kernel upgrade
because i didn't use a loop XFS on this machine before the kernel upgrade.
Can you downgrade your kernel and run the loop device there to tell us whether this is actually a regression or not? If it is a regression, then if you could run a bisect to find the exact patch that causes it woul dbe very helpful....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave!
Quoted 6 lines
I didn't think anything other than log recovery tries to vmap buffers. This is clearly not in log recovery. Can you post an unedited error log, how much data you are rsyncing, the configuration of your filesystem (
xfs_info, mount options, loop dev config, etc) to give us an idea of what you are doing to trigger this?
* I'm attaching the latest kern.log, unedited
* I am syncing the gentoo portage tree, not much data but many small
files (currently 228MiB in 117880 files). This sync is done twice per hour.
* I already posted my xfs_info and mount options in another post. Maybe
i should note that the loop file system was deliberately created with blocksize=512 to accommodate the fs to the nature of the portage tree (many small files...)
* TBH i don't know about any special configuration of the loop device.
I just created an empty file with "dd if=/dev/zero ..." and then did mkfs.xfs on it.
Quoted 1 line
Can't you run on a 64-bit machine?
80% of my machines are 64-bit and i never saw anything like that on them. But otoh i dont' use loop devices very much. Unfortunately this is machine is old hardware (P4 class) which can't run a 64-bit kernel.
Quoted 4 lines
Can you downgrade your kernel and run the loop device there to tell us whether this is actually a regression or not? If it is a regression, then if you could run a bisect to find the exact patch that causes it woul dbe very helpful....
Already did that yesterday and i can confirm it has the same problem - so no regression. The kern.log i attached to this mail is from Kernel 2.6.35.8
cheers,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Michael Weissenbacher
Hi Dave!
Quoted 6 lines
I didn't think anything other than log recovery tries to vmap buffers. This is clearly not in log recovery. Can you post an unedited error log, how much data you are rsyncing, the configuration of your filesystem (
xfs_info, mount options, loop dev config, etc) to give us an idea of what you are doing to trigger this?
Quoted 10 lines
* I'm attaching the latest kern.log, unedited
* I am syncing the gentoo portage tree, not much data but many small
files (currently 228MiB in 117880 files). This sync is done twice per hour.
* I already posted myxfs_infoand mount options in another post. Maybe
i should note that the loop file system was deliberately created with blocksize=512 to accommodate the fs to the nature of the portage tree (many small files...)
* TBH i don't know about any special configuration of the loop device.
I just created an empty file with "dd if=/dev/zero ..." and then did mkfs.xfs on it.
Ok, thanks for the info. I'm struggling to work out what is actually consuming vmap space given what you are doing and the storage config. None of the operations you are doing should be triggering mapping of multipage metadata buffers given the filesystem configuration....
Quoted 5 lines
Quoted 1 line
Can't you run on a 64-bit machine?
80% of my machines are 64-bit and i never saw anything like that on them. But otoh i dont' use loop devices very much. Unfortunately this is machine is old hardware (P4 class) which can't run a 64-bit kernel.
Quoted 7 lines
Quoted 4 lines
Can you downgrade your kernel and run the loop device there to tell us whether this is actually a regression or not? If it is a regression, then if you could run a bisect to find the exact patch that causes it woul dbe very helpful....
Already did that yesterday and i can confirm it has the same problem - so no regression. The kern.log i attached to this mail is from Kernel 2.6.35.8
Ok, I havent looked at this yet, but having looked at the code I don't think it will tell me anything useful. However, if you can reproduce the problem easily, the tracing output (google for trace-cmd and/or kernelshark) of the xfs trace events for the short period across the error condition (just a couple of seconds before and after the error occurs) would tell me a lot more about what is going wrong...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave!
Quoted 7 lines
Ok, thanks for the info. I'm struggling to work out what is actually consuming vmap space given what you are doing and the storage config. None of the operations you are doing should be triggering mapping of multipage metadata buffers given the filesystem configuration....
I've set vmalloc=768M and haven't seen the problem since. Maybe the vmalloc/vmap problem has to do with the fact that i'm running iscsi_target on the same machine? As long as the problem doesn't occur again i'm ok with setting the vmalloc parameter.
Thanks for your help,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave!
Quoted 5 lines · Dave Chinner
Ok, thanks for the info. I'm struggling to work out what is actually consuming vmap space given what you are doing and the storage config. None of the operations you are doing should be triggering mapping of multipage metadata buffers given the filesystem configuration....
Coming back to this, i can confirm that your patch "[PATCH 2/6] vmap:
flush vmap aliases when mapping fails" posted on March 23 2011 did cure my problems.
with kind regards,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs