3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages
While exercising swift on a single node 32bit armhf system running a 3.5 kernel, i got this when i hit ~25% of fs space usage:
dmesg:
...
[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages [ 3037.399485] [ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 [ 3037.399501] [ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) [ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) [ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) [ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) [ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) [ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) [ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c) [ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4) [ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28) [ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c [ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem [ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s) [ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.
flag@c13:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 225G 2.1G 212G 1% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 2.0G 4.0K 2.0G 1% /dev tmpfs 405M 260K 404M 1% /run none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm none 100M 0 100M 0% /run/user /dev/sda1 228M 30M 186M 14% /boot /dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 14958592 74462 14884130 1% / none 182027 1 182026 1% /sys/fs/cgroup udev 177378 1361 176017 1% /dev tmpfs 182027 807 181220 1% /run none 182027 3 182024 1% /run/lock none 182027 1 182026 1% /run/shm none 182027 1 182026 1% /run/user /dev/sda1 124496 35 124461 1% /boot /dev/sda5 524288 237184 287104 46% /mnt/sdb1
the vmalloc space is ~256M usually on this box, so i enlarged it:
flag@c13:~$ dmesg | grep vmalloc
Kernel command line: console=ttyAMA0 nosplash vmalloc=512M
vmalloc : 0xdf800000 - 0xff000000 ( 504 MB)and while i didn't hit the warning above, still after ~25% of usage, the storage node died with:
May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space
left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)
flag@c13:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 225G 3.9G 210G 2% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 2.0G 4.0K 2.0G 1% /dev tmpfs 405M 260K 404M 1% /run none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm none 100M 0 100M 0% /run/user /dev/sda1 228M 25M 192M 12% /boot /dev/sda5 2.0G 564M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 14958592 124409 14834183 1% / none 114542 1 114541 1% /sys/fs/cgroup udev 103895 1361 102534 2% /dev tmpfs 114542 806 113736 1% /run none 114542 3 114539 1% /run/lock none 114542 1 114541 1% /run/shm none 114542 1 114541 1% /run/user /dev/sda1 124496 33 124463 1% /boot /dev/sda5 524288 234880 289408 45% /mnt/sdb1
any idea what else shall i tune to workaround this? or is it a know problem that involves 32bit arch and xfs?
Signature · 7 lines
bye, p. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 89 lines · Paolo Pisati
While exercising swift on a single node 32bit armhf system running a 3.5 kernel, i got this when i hit ~25% of fs space usage:
dmesg:
...[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages [ 3037.399485] [ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 [ 3037.399501] [ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) [ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) [ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) [ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) [ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) [ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) [ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c) [ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4) [ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28) [ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c [ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem [ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s) [ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.flag@c13:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 225G 2.1G 212G 1% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 2.0G 4.0K 2.0G 1% /dev tmpfs 405M 260K 404M 1% /run none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm none 100M 0 100M 0% /run/user /dev/sda1 228M 30M 186M 14% /boot /dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1flag@c13:~$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 14958592 74462 14884130 1% / none 182027 1 182026 1% /sys/fs/cgroup udev 177378 1361 176017 1% /dev tmpfs 182027 807 181220 1% /run none 182027 3 182024 1% /run/lock none 182027 1 182026 1% /run/shm none 182027 1 182026 1% /run/user /dev/sda1 124496 35 124461 1% /boot /dev/sda5 524288 237184 287104 46% /mnt/sdb1the vmalloc space is ~256M usually on this box, so i enlarged it:
flag@c13:~$ dmesg | grep vmalloc Kernel command line: console=ttyAMA0 nosplash vmalloc=512M vmalloc : 0xdf800000 - 0xff000000 ( 504 MB)and while i didn't hit the warning above, still after ~25% of usage, the storage node died with:
May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No spaceleft on device: '
/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)flag@c13:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 225G 3.9G 210G 2% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 2.0G 4.0K 2.0G 1% /dev tmpfs 405M 260K 404M 1% /run none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm none 100M 0 100M 0% /run/user /dev/sda1 228M 25M 192M 12% /boot /dev/sda5 2.0G 564M 1.5G 28% /mnt/sdb1flag@c13:~$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 14958592 124409 14834183 1% / none 114542 1 114541 1% /sys/fs/cgroup udev 103895 1361 102534 2% /dev tmpfs 114542 806 113736 1% /run none 114542 3 114539 1% /run/lock none 114542 1 114541 1% /run/shm none 114542 1 114541 1% /run/user /dev/sda1 124496 33 124463 1% /boot /dev/sda5 524288 234880 289408 45% /mnt/sdb1
Quoted 2 lines
any idea what else shall i tune to workaround this? or is it a know problem that involves 32bit arch and xfs?
I tried to reproduce this issue against the latest upstream tree on 32-bit system but no luck.
Could you please supply the following info:
1) xfs_db -r "-c freesp -s" /dev/sda5
2) xfs_info /mnt/sdb1
Thanks,
-Jeff
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 18 lines · Paolo Pisati
While exercising swift on a single node 32bit armhf system running a 3.5 kernel, i got this when i hit ~25% of fs space usage:
dmesg:
...[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages [ 3037.399485] [ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 [ 3037.399501] [ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) [ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) [ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) [ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) [ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) [ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs])
Quoted 8 lines
[ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c) [ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4) [ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28) [ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c [ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem [ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s) [ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.
Hi Paolo,
You've already contacted me off list about this and pointed me to this:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977
which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug?
Quoted 5 lines
and while i didn't hit the warning above, still after ~25% of usage, the storage node died with:
May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No spaceleft on device: '
/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)
You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096.
I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
As well the freespace info that Jeff asked for?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Dave Chinner
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977
which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug?
the problem with the 'fix' is that it prevents xfs from erroring out, but swift-test fails regardless after ~25% of fs usage and i think having a bold 'xfs error' and a stack trace is more useful.
Quoted 6 lines
You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096.
ok, i repartitioned my disk but i can still reprodue it fairly easily:
df -h:/dev/sda6 216G 573M 215G 1% /mnt/sdb1
df -i:/dev/sda6 56451072 235458 56215614 1% /mnt/sdb1
dmesg:
...
[ 363.130877] XFS (sda6): Mounting Filesystem [ 363.146708] XFS (sda6): Ending clean mount [ 3055.520769] alloc_vmap_area: 18 callbacks suppressed [ 3055.520783] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520817] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520845] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520861] XFS (sda6): xfs_buf_get: failed to map pages [ 3055.520861] [ 3055.520882] XFS (sda6): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 [ 3055.520882] [ 3055.535135] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) [ 3055.535345] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) [ 3055.535687] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) [ 3055.536023] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) [ 3055.536314] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) [ 3055.536589] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8f0>] (xfs_vn_create+0x1c/0x20 [xfs]) [ 3055.536741] [<bf01a8f0>] (xfs_vn_create+0x1c/0x20 [xfs]) from [<c01359d4>] (vfs_create+0xb4/0x120) [ 3055.536760] [<c01359d4>] (vfs_create+0xb4/0x120) from [<c0137c3c>] (do_last+0x860/0x9bc) [ 3055.536775] [<c0137c3c>] (do_last+0x860/0x9bc) from [<c0137fdc>] (path_openat+0xcc/0x428) [ 3055.536787] [<c0137fdc>] (path_openat+0xcc/0x428) from [<c0138458>] (do_filp_open+0x3c/0x90) [ 3055.536805] [<c0138458>] (do_filp_open+0x3c/0x90) from [<c0128248>] (do_sys_open+0xfc/0x1d0) [ 3055.536817] [<c0128248>] (do_sys_open+0xfc/0x1d0) from [<c0128348>] (sys_open+0x2c/0x30) [ 3055.536832] [<c0128348>] (sys_open+0x2c/0x30) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [ 3055.536848] XFS (sda6): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c [ 3055.537327] XFS (sda6): Corruption of in-memory data detected. Shutting down filesystem [ 3055.545439] XFS (sda6): Please umount the filesystem and rectify the problem(s) [ 3070.301048] XFS (sda6): xfs_log_force: error 5 returned. [ 3100.381068] XFS (sda6): xfs_log_force: error 5 returned. [ 3130.461041] XFS (sda6): xfs_log_force: error 5 returned. [ 3160.541042] XFS (sda6): xfs_log_force: error 5 returned. [ 3190.621042] XFS (sda6): xfs_log_force: error 5 returned. [ 3220.701040] XFS (sda6): xfs_log_force: error 5 returned. [ 3250.781039] XFS (sda6): xfs_log_force: error 5 returned. [ 3280.861036] XFS (sda6): xfs_log_force: error 5 returned. [ 3310.941047] XFS (sda6): xfs_log_force: error 5 returned. [ 3341.021044] XFS (sda6): xfs_log_force: error 5 returned. [ 3371.101044] XFS (sda6): xfs_log_force: error 5 returned. [ 3401.181036] XFS (sda6): xfs_log_force: error 5 returned. [ 3431.261036] XFS (sda6): xfs_log_force: error 5 returned. [ 3461.341038] XFS (sda6): xfs_log_force: error 5 returned. [ 3491.421038] XFS (sda6): xfs_log_force: error 5 returned. [ 3521.501051] XFS (sda6): xfs_log_force: error 5 returned. [ 3551.581037] XFS (sda6): xfs_log_force: error 5 returned. [ 3581.661041] XFS (sda6): xfs_log_force: error 5 returned.
Quoted 6 lines
I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
flag@c13:~$ uname -a Linux c13 3.5.0-30-highbank #51-Ubuntu SMP Tue May 14 22:57:15 UTC 2013 armv7l armv7l armv7l GNU/Linux
lag@c13:~$ xfs_repair -V xfs_repair version 3.1.7
armhf highbank node, 4 cores, 4GB mem
flag@c13:~$ cat /proc/meminfo MemTotal: 4137004 kB MemFree: 2719752 kB Buffers: 39688 kB Cached: 580508 kB SwapCached: 0 kB Active: 631136 kB Inactive: 204552 kB Active(anon): 215520 kB Inactive(anon): 232 kB Active(file): 415616 kB Inactive(file): 204320 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 3408896 kB HighFree: 2606516 kB LowTotal: 728108 kB LowFree: 113236 kB SwapTotal: 8378364 kB SwapFree: 8378364 kB Dirty: 4 kB Writeback: 0 kB AnonPages: 215516 kB Mapped: 8676 kB Shmem: 264 kB Slab: 317000 kB SReclaimable: 230392 kB SUnreclaim: 86608 kB KernelStack: 2192 kB PageTables: 2284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10446864 kB Committed_AS: 1049624 kB VmallocTotal: 245760 kB VmallocUsed: 2360 kB VmallocChunk: 241428 kB
flag@c13:~$ cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=2059248k,nr_inodes=177400,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,relatime,size=827404k,mode=755 0 0 /dev/disk/by-uuid/6594b183-3198-4dec-a97d-a3f834b98011 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 /dev/sda1 /boot ext2 rw,relatime,errors=continue 0 0 /dev/sda6 /mnt/sdb1 xfs rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,noquota 0 0
flag@c13:~$ cat /proc/partitions major minor #blocks name
8 0 250059096 sda 8 1 248832 sda1 8 2 15625000 sda2 8 3 1 sda3 8 5 8378369 sda5 8 6 225804288 sda6
no RAID, no LVM
common sata disk
write cache on (unknown size)
no BBWC AFAIK
flag@c13:~$ xfs_info /mnt/sdb1/
meta-data=/dev/sda6 isize=1024 agcount=4, agsize=14112768 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=56451072, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=27564, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0Quoted 2 lines
As well the freespace info that Jeff asked for?
flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6
from to extents blocks pct
1 1 423 423 0.00
2 3 897 2615 0.01
4 7 136 915 0.00
8 15 24833 365797 0.86
8388608 14112768 3 41928421 99.13
total free extents 26292
total free blocks 42298171
average free extent size 1608.78Signature · 7 lines
bye, p. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Paolo Pisati
Quoted 6 lines · Dave Chinner
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977
which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug?
the problem with the 'fix' is that it prevents xfs from erroring out, but swift-test fails regardless after ~25% of fs usage and i think having a bold 'xfs error' and a stack trace is more useful.
I think your logic is misguided. There's a major difference between ENOSPC and a filesystem shutdown. After a shutdown you need to unmount, remount, and then work out what didn't make it to disk before you can restart.
Not to mention that the ENOMEM that triggers the shutdown is highly system dependent - it will occur at different times on different machines and will be highly unpredictable. That's not a good thing.
Compare that to a plain ENOSPC error: you can just remove files and keep going.
Quoted 17 lines
Quoted 6 lines
You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096.
ok, i repartitioned my disk but i can still reprodue it fairly easily:
df -h:
/dev/sda6216G 573M 215G 1%/mnt/sdb1df -i:
/dev/sda656451072 235458 56215614 1%/mnt/sdb1dmesg:
...
Quoted 7 lines
[ 363.130877] XFS (sda6): Mounting Filesystem [ 363.146708] XFS (sda6): Ending clean mount [ 3055.520769] alloc_vmap_area: 18 callbacks suppressed [ 3055.520783] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520817] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520845] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520861] XFS (sda6): xfs_buf_get: failed to map pages
Which is your ENOMEM error, not an ENOSPC error. So the larger filesystem meant you didn't hit the ENOSPC problem like I suspected it would....
Quoted 4 lines
Quoted 4 lines
I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here:
And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....
Quoted 19 lines
Quoted 1 line
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
flag@c13:~$ uname -a Linux c13 3.5.0-30-highbank #51-Ubuntu SMP Tue May 14 22:57:15 UTC 2013 armv7l armv7l armv7l GNU/Linuxlag@c13:~$ xfs_repair -V xfs_repair version 3.1.7armhf highbank node, 4 cores, 4GB mem
flag@c13:~$ cat /proc/meminfo MemTotal: 4137004 kB MemFree: 2719752 kB Buffers: 39688 kB Cached: 580508 kB SwapCached: 0 kB Active: 631136 kB Inactive: 204552 kB Active(anon): 215520 kB
Quoted 8 lines
Inactive(anon): 232 kB Active(file): 415616 kB Inactive(file): 204320 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 3408896 kB HighFree: 2606516 kB LowTotal: 728108 kB
Oh, there's a likely cause of the vmalloc issue. You have 3.4GB of high memory, which means the kernel only has 700MB of low memory for slab caches, vmap regions, etc.
An ia32 box has, by default 960MB of low memory which will be why you are seeing this more frequently than anyone using an ia32 machine. And an ia32 machine can be configured with 2G/2G or 3G/1G kernel/user address space splits, so most vmalloc problems can be worked around.
Quoted 3 lines
Slab: 317000 kB SReclaimable: 230392 kB SUnreclaim: 86608 kB
And so you have 300MB in slab caches in low memory
Quoted 10 lines
KernelStack: 2192 kB PageTables: 2284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10446864 kB Committed_AS: 1049624 kB VmallocTotal: 245760 kB VmallocUsed: 2360 kB VmallocChunk: 241428 kB
and 240MB in vmalloc space. so there's not much left of that 700MB of low memory space. So, you really need that vmap fix, and you need to configure your kernel with more low memory space.
Quoted 12 lines
Quoted 1 line
As well the freespace info that Jeff asked for?
flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6 from to extents blocks pct 1 1 423 423 0.00 2 3 897 2615 0.01 4 7 136 915 0.00 8 15 24833 365797 0.86 8388608 14112768 3 41928421 99.13 total free extents 26292 total free blocks 42298171 average free extent size 1608.78
We need this information after the ENOSPC error occurs, not soon after mkfs or after the ENOMEM error. If this is after ENOSPC, please unmount the filesystem, drop caches and rerun the freesp command...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Dave Chinner
And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....
ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:
dmesg:
...
[163596.605253] updatedb.mlocat: page allocation failure: order:0, mode:0x20 [163596.605299] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04edb20>] (dump_stack+0x20/0x24) [163596.605320] [<c04edb20>] (dump_stack+0x20/0x24) from [<c00e7780>] (warn_alloc_failed+0xd8/0x118) [163596.605335] [<c00e7780>] (warn_alloc_failed+0xd8/0x118) from [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) [163596.605354] [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) from [<c011b798>] (new_slab+0x22c/0x248) [163596.605370] [<c011b798>] (new_slab+0x22c/0x248) from [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) [163596.605383] [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) from [<c011ced4>] (kmem_cache_alloc+0x158/0x190) [163596.605402] [<c011ced4>] (kmem_cache_alloc+0x158/0x190) from [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) [163596.605417] [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) from [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) [163596.605428] [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) from [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) [163596.605439] [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) from [<c0332db0>] (scsi_get_command+0x3c/0xb0) [163596.605453] [<c0332db0>] (scsi_get_command+0x3c/0xb0) from [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) [163596.605466] [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) from [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac) [163596.605482] [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac) from [<c0343568>] (sd_prep_fn+0x114/0xaf4) [163596.605501] [<c0343568>] (sd_prep_fn+0x114/0xaf4) from [<c0299af4>] (blk_peek_request+0xc8/0x214) [163596.605514] [<c0299af4>] (blk_peek_request+0xc8/0x214) from [<c033a1b0>] (scsi_request_fn+0x40/0x504) [163596.605524] [<c033a1b0>] (scsi_request_fn+0x40/0x504) from [<c029a38c>] (blk_queue_bio+0x300/0x384) [163596.605536] [<c029a38c>] (blk_queue_bio+0x300/0x384) from [<c0298450>] (generic_make_request+0xb8/0xd8) [163596.605548] [<c0298450>] (generic_make_request+0xb8/0xd8) from [<c0298534>] (submit_bio+0xc4/0x17c) [163596.605756] [<c0298534>] (submit_bio+0xc4/0x17c) from [<bf00f1c4>] (_xfs_buf_ioapply+0x1bc/0x224 [xfs]) [163596.606002] [<bf00f1c4>] (_xfs_buf_ioapply+0x1bc/0x224 [xfs]) from [<bf00f314>] (xfs_buf_iorequest+0x4c/0x98 [xfs]) [163596.606241] [<bf00f314>] (xfs_buf_iorequest+0x4c/0x98 [xfs]) from [<bf00f868>] (_xfs_buf_read+0x34/0x50 [xfs]) [163596.606481] [<bf00f868>] (_xfs_buf_read+0x34/0x50 [xfs]) from [<bf00f964>] (xfs_buf_read+0xe0/0x108 [xfs]) [163596.606781] [<bf00f964>] (xfs_buf_read+0xe0/0x108 [xfs]) from [<bf06ba78>] (xfs_trans_read_buf+0x1e4/0x3e8 [xfs]) [163596.607115] [<bf06ba78>] (xfs_trans_read_buf+0x1e4/0x3e8 [xfs]) from [<bf053a9c>] (xfs_imap_to_bp+0x54/0x128 [xfs]) [163596.607432] [<bf053a9c>] (xfs_imap_to_bp+0x54/0x128 [xfs]) from [<bf057bc4>] (xfs_iread+0x6c/0x150 [xfs]) [163596.607719] [<bf057bc4>] (xfs_iread+0x6c/0x150 [xfs]) from [<bf015bfc>] (xfs_iget+0x210/0x72c [xfs]) [163596.607982] [<bf015bfc>] (xfs_iget+0x210/0x72c [xfs]) from [<bf0233b4>] (xfs_lookup+0xf4/0x114 [xfs]) [163596.608247] [<bf0233b4>] (xfs_lookup+0xf4/0x114 [xfs]) from [<bf01a5e8>] (xfs_vn_lookup+0x54/0x98 [xfs]) [163596.608387] [<bf01a5e8>] (xfs_vn_lookup+0x54/0x98 [xfs]) from [<c0134198>] (__lookup_hash+0x64/0xec) [163596.608402] [<c0134198>] (__lookup_hash+0x64/0xec) from [<c04f0d68>] (lookup_slow+0x50/0xac) [163596.608415] [<c04f0d68>] (lookup_slow+0x50/0xac) from [<c0136724>] (path_lookupat+0x730/0x794) [163596.608428] [<c0136724>] (path_lookupat+0x730/0x794) from [<c01367b4>] (do_path_lookup+0x2c/0xd0) [163596.608439] [<c01367b4>] (do_path_lookup+0x2c/0xd0) from [<c01385e0>] (user_path_at_empty+0x64/0x8c) [163596.608451] [<c01385e0>] (user_path_at_empty+0x64/0x8c) from [<c013862c>] (user_path_at+0x24/0x2c) [163596.608462] [<c013862c>] (user_path_at+0x24/0x2c) from [<c012dd3c>] (vfs_fstatat+0x40/0x78) [163596.608473] [<c012dd3c>] (vfs_fstatat+0x40/0x78) from [<c012dd9c>] (vfs_lstat+0x28/0x2c) [163596.608482] [<c012dd9c>] (vfs_lstat+0x28/0x2c) from [<c012e048>] (sys_lstat64+0x24/0x40) [163596.608495] [<c012e048>] (sys_lstat64+0x24/0x40) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [163596.608503] Mem-info: [163596.608509] Normal per-cpu: [163596.608515] CPU 0: hi: 186, btch: 31 usd: 38 [163596.608521] CPU 1: hi: 186, btch: 31 usd: 218 [163596.608528] CPU 2: hi: 186, btch: 31 usd: 152 [163596.608533] CPU 3: hi: 186, btch: 31 usd: 171 [163596.608538] HighMem per-cpu: [163596.608544] CPU 0: hi: 186, btch: 31 usd: 46 [163596.608549] CPU 1: hi: 186, btch: 31 usd: 171 [163596.608555] CPU 2: hi: 186, btch: 31 usd: 168 [163596.608561] CPU 3: hi: 186, btch: 31 usd: 177 [163596.608574] active_anon:26367 inactive_anon:29153 isolated_anon:0 [163596.608574] active_file:396338 inactive_file:397959 isolated_file:0 [163596.608574] unevictable:0 dirty:0 writeback:5 unstable:0 [163596.608574] free:5145 slab_reclaimable:57625 slab_unreclaimable:7729 [163596.608574] mapped:1703 shmem:10 pagetables:581 bounce:0 [163596.608602] Normal free:15256kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:8kB active_file:848kB inactive_file:1560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:230500kB slab_unreclaimable:30916kB kernel_stack:2208kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [163596.608607] lowmem_reserve[]: 0 26423 26423 [163596.608628] HighMem free:5324kB min:512kB low:4352kB high:8192kB active_anon:105468kB inactive_anon:116604kB active_file:1584504kB inactive_file:1590276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:20kB mapped:6812kB shmem:40kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [163596.608634] lowmem_reserve[]: 0 0 0 [163596.608643] Normal: 216*4kB 215*8kB 216*16kB 216*32kB 36*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15256kB [163596.608668] HighMem: 233*4kB 67*8kB 141*16kB 22*32kB 8*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5324kB [163596.608692] 794329 total pagecache pages [163596.608697] 12 pages in swap cache [163596.608703] Swap cache stats: add 79, delete 67, find 9/11 [163596.608708] Free swap = 8378092kB [163596.608712] Total swap = 8378364kB [163596.670667] 1046784 pages of RAM [163596.670674] 6801 free pages [163596.670679] 12533 reserved pages [163596.670683] 36489 slab pages [163596.670687] 631668 pages shared [163596.670692] 12 pages swap cached [163596.670701] SLUB: Unable to allocate memory on node -1 (gfp=0x8020) [163596.670710] cache: kmalloc-192, object size: 192, buffer size: 192, default order: 0, min order: 0 [163596.670718] node 0: slabs: 2733, objs: 57393, free: 0
df -h:
.../dev/sda6 216G 53G 163G 25% /mnt/sdb1
df -i:
.../dev/sda6 56451072 19721920 36729152 35% /mnt/sdb1
flag@c13:~$ cat /proc/meminfo MemTotal: 4137004 kB MemFree: 1191096 kB Buffers: 23172 kB Cached: 2074116 kB SwapCached: 48 kB Active: 1301568 kB Inactive: 1016024 kB Active(anon): 103748 kB Inactive(anon): 116600 kB Active(file): 1197820 kB Inactive(file): 899424 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 3408896 kB HighFree: 1108444 kB LowTotal: 728108 kB LowFree: 82652 kB SwapTotal: 8378364 kB SwapFree: 8378092 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 220284 kB Mapped: 7068 kB Shmem: 44 kB Slab: 263216 kB SReclaimable: 232212 kB SUnreclaim: 31004 kB KernelStack: 2192 kB PageTables: 2312 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10446864 kB Committed_AS: 1051868 kB VmallocTotal: 245760 kB VmallocUsed: 2360 kB VmallocChunk: 241428 kB
flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6
from to extents blocks pct
1 1 27058 27058 0.06
2 3 124367 358831 0.84
4 7 17656 121693 0.29
8 15 2856900 42122381 98.81
total free extents 3025981
total free blocks 42629963
average free extent size 14.088Signature · 7 lines
bye, p. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Paolo Pisati
Quoted 3 lines · Dave Chinner
And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....
ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:
i'm testing a 3.5.y kernel plus those 3 patches:
549142a xfs: don't use speculative prealloc for small files f0843f4 xfs: limit speculative prealloc size on sparse files 454da09 xfs: inode allocation should use unmapped buffers.
and i can confirm that:
-using a small fs (2G) i cannot reproduce any -ENOSPC or vmalloc() problem anymore, the benchmark runs until running out of inodes
-using a bigger fs (~250G), two days and my tests are still running good
Signature · 7 lines
bye, p. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 16 lines · Paolo Pisati
Quoted 8 lines · Paolo Pisati
Quoted 3 lines · Dave Chinner
And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....
ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:
i'm testing a 3.5.y kernel plus those 3 patches:
549142a xfs: don't use speculative prealloc for small files f0843f4 xfs: limit speculative prealloc size on sparse files 454da09 xfs: inode allocation should use unmapped buffers.
Quoted 6 lines
and i can confirm that:
-using a small fs (2G) i cannot reproduce any
-ENOSPCorvmalloc()problem anymore, the benchmark runs until running out of inodes-using a bigger fs (~250G), two days and my tests are still running good
Ok, good to know. The first patch you list there hasn't even been reviewed yet, so it might take some time before that is ready for -stable backport.
Also, there are a bunch of fixes needed to the second patch you have there (f0843f4 xfs: limit speculative prealloc...) that would also be necessary for a -stable backport. i.e:
e8108ce xfs: fix xfs_iomap_eof_prealloc_initial_size type e114b5f xfs: increase prealloc size to double that of the previous extent e78c420 xfs: fix potential infinite loop in xfs_iomap_prealloc_size()
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Paolo Pisati
Quoted 3 lines · Dave Chinner
And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....
ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:
dmesg:
...
Quoted 13 lines
[163596.605253] updatedb.mlocat: page allocation failure: order:0, mode:0x20 [163596.605299] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04edb20>] (dump_stack+0x20/0x24) [163596.605320] [<c04edb20>] (dump_stack+0x20/0x24) from [<c00e7780>] (warn_alloc_failed+0xd8/0x118) [163596.605335] [<c00e7780>] (warn_alloc_failed+0xd8/0x118) from [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) [163596.605354] [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) from [<c011b798>] (new_slab+0x22c/0x248) [163596.605370] [<c011b798>] (new_slab+0x22c/0x248) from [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) [163596.605383] [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) from [<c011ced4>] (kmem_cache_alloc+0x158/0x190) [163596.605402] [<c011ced4>] (kmem_cache_alloc+0x158/0x190) from [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) [163596.605417] [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) from [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) [163596.605428] [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) from [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) [163596.605439] [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) from [<c0332db0>] (scsi_get_command+0x3c/0xb0) [163596.605453] [<c0332db0>] (scsi_get_command+0x3c/0xb0) from [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) [163596.605466] [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) from [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac)
ENOMEM deep in the SCSI stack for an order 0 GFP_ATOMIC allocation. That's not an XFS problem - that's a SCSI stack issue. You should probably report that to the scsi list...
Quoted 11 lines
[163596.608574] active_anon:26367 inactive_anon:29153 isolated_anon:0 [163596.608574] active_file:396338 inactive_file:397959 isolated_file:0 [163596.608574] unevictable:0 dirty:0 writeback:5 unstable:0 [163596.608574] free:5145 slab_reclaimable:57625 slab_unreclaimable:7729 [163596.608574] mapped:1703 shmem:10 pagetables:581 bounce:0 [163596.608602] Normal free:15256kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:8kB active_file:848kB inactive_file:1560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:230500kB slab_unreclaimable:30916kB kernel_stack:2208kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [163596.608607] lowmem_reserve[]: 0 26423 26423 [163596.608628] HighMem free:5324kB min:512kB low:4352kB high:8192kB active_anon:105468kB inactive_anon:116604kB active_file:1584504kB inactive_file:1590276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:20kB mapped:6812kB shmem:40kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [163596.608634] lowmem_reserve[]: 0 0 0 [163596.608643] Normal: 216*4kB 215*8kB 216*16kB 216*32kB 36*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15256kB [163596.608668] HighMem: 233*4kB 67*8kB 141*16kB 22*32kB 8*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5324kB
Though this says there is plenty of free order 0 pages in both low and high memory....
Quoted 6 lines
[163596.608692] 794329 total pagecache pages [163596.608697] 12 pages in swap cache [163596.608703] Swap cache stats: add 79, delete 67, find 9/11 [163596.608708] Free swap = 8378092kB [163596.608712] Total swap = 8378364kB [163596.670667] 1046784 pages of RAM
Quoted 8 lines
[163596.670674] 6801 free pages [163596.670679] 12533 reserved pages [163596.670683] 36489 slab pages [163596.670687] 631668 pages shared [163596.670692] 12 pages swap cached [163596.670701] SLUB: Unable to allocate memory on node -1 (gfp=0x8020) [163596.670710] cache: kmalloc-192, object size: 192, buffer size: 192, default order: 0, min order: 0 [163596.670718] node 0: slabs: 2733, objs: 57393, free: 0
And it was slub that was unable to find a page when it should have been able to, so perhaps this is a VM problem?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs