Skip to contentSkip to conversation

3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages

DISCUSSION Paolo Pisati 3 participants 9 messages May 17, 2013 – May 30, 2013

lore
Share & tools
lore
#0 lore raw

While exercising swift on a single node 32bit armhf system running a 3.5 kernel, i got this when i hit ~25% of fs space usage:

dmesg:
...

[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages
[ 3037.399485]
[ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0
[ 3037.399501]
[ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24)
[ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs])
[ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs])
[ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs])
[ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs])
[ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs])
[ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c)
[ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4)
[ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28)
[ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30)
[ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c
[ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem
[ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s)
[ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.
flag@c13:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 225G 2.1G 212G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 405M 260K 404M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 228M 30M 186M 14% /boot
/dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 14958592 74462 14884130 1% /
none 182027 1 182026 1% /sys/fs/cgroup
udev 177378 1361 176017 1% /dev
tmpfs 182027 807 181220 1% /run
none 182027 3 182024 1% /run/lock
none 182027 1 182026 1% /run/shm
none 182027 1 182026 1% /run/user
/dev/sda1 124496 35 124461 1% /boot
/dev/sda5 524288 237184 287104 46% /mnt/sdb1

the vmalloc space is ~256M usually on this box, so i enlarged it:

flag@c13:~$ dmesg | grep vmalloc                                                                                                                                                          
Kernel command line: console=ttyAMA0 nosplash vmalloc=512M                                                                                                                                
    vmalloc : 0xdf800000 - 0xff000000   ( 504 MB)

and while i didn't hit the warning above, still after ~25% of usage, the storage node died with:

May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space

left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)

flag@c13:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       225G  3.9G  210G   2% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           405M  260K  404M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       228M   25M  192M  12% /boot
/dev/sda5       2.0G  564M  1.5G  28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda2      14958592 124409 14834183    1% /
none             114542      1   114541    1% /sys/fs/cgroup
udev             103895   1361   102534    2% /dev
tmpfs            114542    806   113736    1% /run
none             114542      3   114539    1% /run/lock
none             114542      1   114541    1% /run/shm
none             114542      1   114541    1% /run/user
/dev/sda1        124496     33   124463    1% /boot
/dev/sda5        524288 234880   289408   45% /mnt/sdb1

any idea what else shall i tune to workaround this? or is it a know problem that involves 32bit arch and xfs?

Signature · 7 lines
bye,
p.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 89 lines · Paolo Pisati

While exercising swift on a single node 32bit armhf system running a 3.5 kernel, i got this when i hit ~25% of fs space usage:

dmesg:
...

[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages
[ 3037.399485]
[ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0
[ 3037.399501]
[ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24)
[ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs])
[ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs])
[ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs])
[ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs])
[ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs])
[ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c)
[ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4)
[ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28)
[ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30)
[ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c
[ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem
[ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s)
[ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.
flag@c13:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 225G 2.1G 212G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 405M 260K 404M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 228M 30M 186M 14% /boot
/dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 14958592 74462 14884130 1% /
none 182027 1 182026 1% /sys/fs/cgroup
udev 177378 1361 176017 1% /dev
tmpfs 182027 807 181220 1% /run
none 182027 3 182024 1% /run/lock
none 182027 1 182026 1% /run/shm
none 182027 1 182026 1% /run/user
/dev/sda1 124496 35 124461 1% /boot
/dev/sda5 524288 237184 287104 46% /mnt/sdb1

the vmalloc space is ~256M usually on this box, so i enlarged it:

flag@c13:~$ dmesg | grep vmalloc                                                                                                                                                          
Kernel command line: console=ttyAMA0 nosplash vmalloc=512M                                                                                                                                
    vmalloc : 0xdf800000 - 0xff000000   ( 504 MB)

and while i didn't hit the warning above, still after ~25% of usage, the storage node died with:

May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space

left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)

flag@c13:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       225G  3.9G  210G   2% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           405M  260K  404M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       228M   25M  192M  12% /boot
/dev/sda5       2.0G  564M  1.5G  28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda2      14958592 124409 14834183    1% /
none             114542      1   114541    1% /sys/fs/cgroup
udev             103895   1361   102534    2% /dev
tmpfs            114542    806   113736    1% /run
none             114542      3   114539    1% /run/lock
none             114542      1   114541    1% /run/shm
none             114542      1   114541    1% /run/user
/dev/sda1        124496     33   124463    1% /boot
/dev/sda5        524288 234880   289408   45% /mnt/sdb1
Quoted 2 lines

any idea what else shall i tune to workaround this? or is it a know problem that involves 32bit arch and xfs?

I tried to reproduce this issue against the latest upstream tree on 32-bit system but no luck.

Could you please supply the following info:

1) xfs_db -r "-c freesp -s" /dev/sda5
2) xfs_info /mnt/sdb1

Thanks,
-Jeff

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
In reply to Paolo Pisati
Quoted 18 lines · Paolo Pisati

While exercising swift on a single node 32bit armhf system running a 3.5 kernel, i got this when i hit ~25% of fs space usage:

dmesg:
...

[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages
[ 3037.399485]
[ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0
[ 3037.399501]
[ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24)
[ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs])
[ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs])
[ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs])
[ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs])
[ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs])
Quoted 8 lines
[ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c)
[ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4)
[ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28)
[ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30)
[ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c
[ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem
[ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s)
[ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.

Hi Paolo,

You've already contacted me off list about this and pointed me to this:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977

which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug?

Quoted 5 lines

and while i didn't hit the warning above, still after ~25% of usage, the storage node died with:

May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space

left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)

You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096.

I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

As well the freespace info that Jeff asked for?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 6 lines · Dave Chinner

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977

which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug?

the problem with the 'fix' is that it prevents xfs from erroring out, but swift-test fails regardless after ~25% of fs usage and i think having a bold 'xfs error' and a stack trace is more useful.

Quoted 6 lines

You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096.

ok, i repartitioned my disk but i can still reprodue it fairly easily:

df -h:
/dev/sda6 216G 573M 215G 1% /mnt/sdb1

df -i:
/dev/sda6 56451072 235458 56215614 1% /mnt/sdb1

dmesg:
...

[  363.130877] XFS (sda6): Mounting Filesystem
[  363.146708] XFS (sda6): Ending clean mount
[ 3055.520769] alloc_vmap_area: 18 callbacks suppressed
[ 3055.520783] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3055.520817] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3055.520845] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3055.520861] XFS (sda6): xfs_buf_get: failed to map pages
[ 3055.520861]
[ 3055.520882] XFS (sda6): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c.  Caller 0xbf0235e0
[ 3055.520882]
[ 3055.535135] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24)
[ 3055.535345] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs])
[ 3055.535687] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs])
[ 3055.536023] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs])
[ 3055.536314] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs])
[ 3055.536589] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8f0>] (xfs_vn_create+0x1c/0x20 [xfs])
[ 3055.536741] [<bf01a8f0>] (xfs_vn_create+0x1c/0x20 [xfs]) from [<c01359d4>] (vfs_create+0xb4/0x120)
[ 3055.536760] [<c01359d4>] (vfs_create+0xb4/0x120) from [<c0137c3c>] (do_last+0x860/0x9bc)
[ 3055.536775] [<c0137c3c>] (do_last+0x860/0x9bc) from [<c0137fdc>] (path_openat+0xcc/0x428)
[ 3055.536787] [<c0137fdc>] (path_openat+0xcc/0x428) from [<c0138458>] (do_filp_open+0x3c/0x90)
[ 3055.536805] [<c0138458>] (do_filp_open+0x3c/0x90) from [<c0128248>] (do_sys_open+0xfc/0x1d0)
[ 3055.536817] [<c0128248>] (do_sys_open+0xfc/0x1d0) from [<c0128348>] (sys_open+0x2c/0x30)
[ 3055.536832] [<c0128348>] (sys_open+0x2c/0x30) from [<c000e320>] (ret_fast_syscall+0x0/0x30)
[ 3055.536848] XFS (sda6): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c.  Return address = 0xbf06340c
[ 3055.537327] XFS (sda6): Corruption of in-memory data detected.  Shutting down filesystem
[ 3055.545439] XFS (sda6): Please umount the filesystem and rectify the problem(s)
[ 3070.301048] XFS (sda6): xfs_log_force: error 5 returned.
[ 3100.381068] XFS (sda6): xfs_log_force: error 5 returned.
[ 3130.461041] XFS (sda6): xfs_log_force: error 5 returned.
[ 3160.541042] XFS (sda6): xfs_log_force: error 5 returned.
[ 3190.621042] XFS (sda6): xfs_log_force: error 5 returned.
[ 3220.701040] XFS (sda6): xfs_log_force: error 5 returned.
[ 3250.781039] XFS (sda6): xfs_log_force: error 5 returned.
[ 3280.861036] XFS (sda6): xfs_log_force: error 5 returned.
[ 3310.941047] XFS (sda6): xfs_log_force: error 5 returned.
[ 3341.021044] XFS (sda6): xfs_log_force: error 5 returned.
[ 3371.101044] XFS (sda6): xfs_log_force: error 5 returned.
[ 3401.181036] XFS (sda6): xfs_log_force: error 5 returned.
[ 3431.261036] XFS (sda6): xfs_log_force: error 5 returned.
[ 3461.341038] XFS (sda6): xfs_log_force: error 5 returned.
[ 3491.421038] XFS (sda6): xfs_log_force: error 5 returned.
[ 3521.501051] XFS (sda6): xfs_log_force: error 5 returned.
[ 3551.581037] XFS (sda6): xfs_log_force: error 5 returned.
[ 3581.661041] XFS (sda6): xfs_log_force: error 5 returned.
Quoted 6 lines

I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

flag@c13:~$ uname -a
Linux c13 3.5.0-30-highbank #51-Ubuntu SMP Tue May 14 22:57:15 UTC 2013 armv7l armv7l armv7l GNU/Linux
lag@c13:~$ xfs_repair -V
xfs_repair version 3.1.7

armhf highbank node, 4 cores, 4GB mem

flag@c13:~$ cat /proc/meminfo
MemTotal:        4137004 kB
MemFree:         2719752 kB
Buffers:           39688 kB
Cached:           580508 kB
SwapCached:            0 kB
Active:           631136 kB
Inactive:         204552 kB
Active(anon):     215520 kB
Inactive(anon):      232 kB
Active(file):     415616 kB
Inactive(file):   204320 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       3408896 kB
HighFree:        2606516 kB
LowTotal:         728108 kB
LowFree:          113236 kB
SwapTotal:       8378364 kB
SwapFree:        8378364 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:        215516 kB
Mapped:             8676 kB
Shmem:               264 kB
Slab:             317000 kB
SReclaimable:     230392 kB
SUnreclaim:        86608 kB
KernelStack:        2192 kB
PageTables:         2284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10446864 kB
Committed_AS:    1049624 kB
VmallocTotal:     245760 kB
VmallocUsed:        2360 kB
VmallocChunk:     241428 kB
flag@c13:~$ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=2059248k,nr_inodes=177400,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=827404k,mode=755 0 0
/dev/disk/by-uuid/6594b183-3198-4dec-a97d-a3f834b98011 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
/dev/sda1 /boot ext2 rw,relatime,errors=continue 0 0
/dev/sda6 /mnt/sdb1 xfs rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,noquota 0 0
flag@c13:~$ cat /proc/partitions
major minor  #blocks  name
   8        0  250059096 sda
   8        1     248832 sda1
   8        2   15625000 sda2
   8        3          1 sda3
   8        5    8378369 sda5
   8        6  225804288 sda6

no RAID, no LVM

common sata disk

write cache on (unknown size)

no BBWC AFAIK

flag@c13:~$ xfs_info /mnt/sdb1/
meta-data=/dev/sda6              isize=1024   agcount=4, agsize=14112768 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=56451072, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=27564, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Quoted 2 lines

As well the freespace info that Jeff asked for?

flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6 
   from      to extents  blocks    pct
      1       1     423     423   0.00
      2       3     897    2615   0.01
      4       7     136     915   0.00
      8      15   24833  365797   0.86
8388608 14112768       3 41928421  99.13
total free extents 26292
total free blocks 42298171
average free extent size 1608.78
Signature · 7 lines
bye,
p.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 11 lines · Paolo Pisati
Quoted 6 lines · Dave Chinner

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977

which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug?

the problem with the 'fix' is that it prevents xfs from erroring out, but swift-test fails regardless after ~25% of fs usage and i think having a bold 'xfs error' and a stack trace is more useful.

I think your logic is misguided. There's a major difference between ENOSPC and a filesystem shutdown. After a shutdown you need to unmount, remount, and then work out what didn't make it to disk before you can restart.

Not to mention that the ENOMEM that triggers the shutdown is highly system dependent - it will occur at different times on different machines and will be highly unpredictable. That's not a good thing.

Compare that to a plain ENOSPC error: you can just remove files and keep going.

Quoted 17 lines
Quoted 6 lines

You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096.

ok, i repartitioned my disk but i can still reprodue it fairly easily:

df -h:
/dev/sda6 216G 573M 215G 1% /mnt/sdb1

df -i:
/dev/sda6 56451072 235458 56215614 1% /mnt/sdb1

dmesg:
...

Quoted 7 lines
[  363.130877] XFS (sda6): Mounting Filesystem
[  363.146708] XFS (sda6): Ending clean mount
[ 3055.520769] alloc_vmap_area: 18 callbacks suppressed
[ 3055.520783] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3055.520817] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3055.520845] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3055.520861] XFS (sda6): xfs_buf_get: failed to map pages

Which is your ENOMEM error, not an ENOSPC error. So the larger filesystem meant you didn't hit the ENOSPC problem like I suspected it would....

Quoted 4 lines
Quoted 4 lines

I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here:

And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....

Quoted 19 lines
Quoted 1 line

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

flag@c13:~$ uname -a
Linux c13 3.5.0-30-highbank #51-Ubuntu SMP Tue May 14 22:57:15 UTC 2013 armv7l armv7l armv7l GNU/Linux
lag@c13:~$ xfs_repair -V
xfs_repair version 3.1.7

armhf highbank node, 4 cores, 4GB mem

flag@c13:~$ cat /proc/meminfo
MemTotal:        4137004 kB
MemFree:         2719752 kB
Buffers:           39688 kB
Cached:           580508 kB
SwapCached:            0 kB
Active:           631136 kB
Inactive:         204552 kB
Active(anon):     215520 kB
Quoted 8 lines
Inactive(anon):      232 kB
Active(file):     415616 kB
Inactive(file):   204320 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       3408896 kB
HighFree:        2606516 kB
LowTotal:         728108 kB

Oh, there's a likely cause of the vmalloc issue. You have 3.4GB of high memory, which means the kernel only has 700MB of low memory for slab caches, vmap regions, etc.

An ia32 box has, by default 960MB of low memory which will be why you are seeing this more frequently than anyone using an ia32 machine. And an ia32 machine can be configured with 2G/2G or 3G/1G kernel/user address space splits, so most vmalloc problems can be worked around.

Quoted 3 lines
Slab:             317000 kB
SReclaimable:     230392 kB
SUnreclaim:        86608 kB

And so you have 300MB in slab caches in low memory

Quoted 10 lines
KernelStack:        2192 kB
PageTables:         2284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10446864 kB
Committed_AS:    1049624 kB
VmallocTotal:     245760 kB
VmallocUsed:        2360 kB
VmallocChunk:     241428 kB

and 240MB in vmalloc space. so there's not much left of that 700MB of low memory space. So, you really need that vmap fix, and you need to configure your kernel with more low memory space.

Quoted 12 lines
Quoted 1 line

As well the freespace info that Jeff asked for?

flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6 
   from      to extents  blocks    pct
      1       1     423     423   0.00
      2       3     897    2615   0.01
      4       7     136     915   0.00
      8      15   24833  365797   0.86
8388608 14112768       3 41928421  99.13
total free extents 26292
total free blocks 42298171
average free extent size 1608.78

We need this information after the ENOSPC error occurs, not soon after mkfs or after the ENOMEM error. If this is after ENOSPC, please unmount the filesystem, drop caches and rerun the freesp command...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
Quoted 3 lines · Dave Chinner

And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....

ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:

dmesg:
...

[163596.605253] updatedb.mlocat: page allocation failure: order:0, mode:0x20
[163596.605299] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04edb20>] (dump_stack+0x20/0x24)
[163596.605320] [<c04edb20>] (dump_stack+0x20/0x24) from [<c00e7780>] (warn_alloc_failed+0xd8/0x118)
[163596.605335] [<c00e7780>] (warn_alloc_failed+0xd8/0x118) from [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708)
[163596.605354] [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) from [<c011b798>] (new_slab+0x22c/0x248)
[163596.605370] [<c011b798>] (new_slab+0x22c/0x248) from [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8)
[163596.605383] [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) from [<c011ced4>] (kmem_cache_alloc+0x158/0x190)
[163596.605402] [<c011ced4>] (kmem_cache_alloc+0x158/0x190) from [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74)
[163596.605417] [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) from [<c0332c80>] (scsi_host_alloc_command+0x24/0x78)
[163596.605428] [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) from [<c0332cf0>] (__scsi_get_command+0x1c/0xa0)
[163596.605439] [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) from [<c0332db0>] (scsi_get_command+0x3c/0xb0)
[163596.605453] [<c0332db0>] (scsi_get_command+0x3c/0xb0) from [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60)
[163596.605466] [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) from [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac)
[163596.605482] [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac) from [<c0343568>] (sd_prep_fn+0x114/0xaf4)
[163596.605501] [<c0343568>] (sd_prep_fn+0x114/0xaf4) from [<c0299af4>] (blk_peek_request+0xc8/0x214)
[163596.605514] [<c0299af4>] (blk_peek_request+0xc8/0x214) from [<c033a1b0>] (scsi_request_fn+0x40/0x504)
[163596.605524] [<c033a1b0>] (scsi_request_fn+0x40/0x504) from [<c029a38c>] (blk_queue_bio+0x300/0x384)
[163596.605536] [<c029a38c>] (blk_queue_bio+0x300/0x384) from [<c0298450>] (generic_make_request+0xb8/0xd8)
[163596.605548] [<c0298450>] (generic_make_request+0xb8/0xd8) from [<c0298534>] (submit_bio+0xc4/0x17c)
[163596.605756] [<c0298534>] (submit_bio+0xc4/0x17c) from [<bf00f1c4>] (_xfs_buf_ioapply+0x1bc/0x224 [xfs])
[163596.606002] [<bf00f1c4>] (_xfs_buf_ioapply+0x1bc/0x224 [xfs]) from [<bf00f314>] (xfs_buf_iorequest+0x4c/0x98 [xfs])
[163596.606241] [<bf00f314>] (xfs_buf_iorequest+0x4c/0x98 [xfs]) from [<bf00f868>] (_xfs_buf_read+0x34/0x50 [xfs])
[163596.606481] [<bf00f868>] (_xfs_buf_read+0x34/0x50 [xfs]) from [<bf00f964>] (xfs_buf_read+0xe0/0x108 [xfs])
[163596.606781] [<bf00f964>] (xfs_buf_read+0xe0/0x108 [xfs]) from [<bf06ba78>] (xfs_trans_read_buf+0x1e4/0x3e8 [xfs])
[163596.607115] [<bf06ba78>] (xfs_trans_read_buf+0x1e4/0x3e8 [xfs]) from [<bf053a9c>] (xfs_imap_to_bp+0x54/0x128 [xfs])
[163596.607432] [<bf053a9c>] (xfs_imap_to_bp+0x54/0x128 [xfs]) from [<bf057bc4>] (xfs_iread+0x6c/0x150 [xfs])
[163596.607719] [<bf057bc4>] (xfs_iread+0x6c/0x150 [xfs]) from [<bf015bfc>] (xfs_iget+0x210/0x72c [xfs])
[163596.607982] [<bf015bfc>] (xfs_iget+0x210/0x72c [xfs]) from [<bf0233b4>] (xfs_lookup+0xf4/0x114 [xfs])
[163596.608247] [<bf0233b4>] (xfs_lookup+0xf4/0x114 [xfs]) from [<bf01a5e8>] (xfs_vn_lookup+0x54/0x98 [xfs])
[163596.608387] [<bf01a5e8>] (xfs_vn_lookup+0x54/0x98 [xfs]) from [<c0134198>] (__lookup_hash+0x64/0xec)
[163596.608402] [<c0134198>] (__lookup_hash+0x64/0xec) from [<c04f0d68>] (lookup_slow+0x50/0xac)
[163596.608415] [<c04f0d68>] (lookup_slow+0x50/0xac) from [<c0136724>] (path_lookupat+0x730/0x794)
[163596.608428] [<c0136724>] (path_lookupat+0x730/0x794) from [<c01367b4>] (do_path_lookup+0x2c/0xd0)
[163596.608439] [<c01367b4>] (do_path_lookup+0x2c/0xd0) from [<c01385e0>] (user_path_at_empty+0x64/0x8c)
[163596.608451] [<c01385e0>] (user_path_at_empty+0x64/0x8c) from [<c013862c>] (user_path_at+0x24/0x2c)
[163596.608462] [<c013862c>] (user_path_at+0x24/0x2c) from [<c012dd3c>] (vfs_fstatat+0x40/0x78)
[163596.608473] [<c012dd3c>] (vfs_fstatat+0x40/0x78) from [<c012dd9c>] (vfs_lstat+0x28/0x2c)
[163596.608482] [<c012dd9c>] (vfs_lstat+0x28/0x2c) from [<c012e048>] (sys_lstat64+0x24/0x40)
[163596.608495] [<c012e048>] (sys_lstat64+0x24/0x40) from [<c000e320>] (ret_fast_syscall+0x0/0x30)
[163596.608503] Mem-info:
[163596.608509] Normal per-cpu:
[163596.608515] CPU    0: hi:  186, btch:  31 usd:  38
[163596.608521] CPU    1: hi:  186, btch:  31 usd: 218
[163596.608528] CPU    2: hi:  186, btch:  31 usd: 152
[163596.608533] CPU    3: hi:  186, btch:  31 usd: 171
[163596.608538] HighMem per-cpu:
[163596.608544] CPU    0: hi:  186, btch:  31 usd:  46
[163596.608549] CPU    1: hi:  186, btch:  31 usd: 171
[163596.608555] CPU    2: hi:  186, btch:  31 usd: 168
[163596.608561] CPU    3: hi:  186, btch:  31 usd: 177
[163596.608574] active_anon:26367 inactive_anon:29153 isolated_anon:0
[163596.608574]  active_file:396338 inactive_file:397959 isolated_file:0
[163596.608574]  unevictable:0 dirty:0 writeback:5 unstable:0
[163596.608574]  free:5145 slab_reclaimable:57625 slab_unreclaimable:7729
[163596.608574]  mapped:1703 shmem:10 pagetables:581 bounce:0
[163596.608602] Normal free:15256kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:8kB active_file:848kB inactive_file:1560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:230500kB slab_unreclaimable:30916kB kernel_stack:2208kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[163596.608607] lowmem_reserve[]: 0 26423 26423
[163596.608628] HighMem free:5324kB min:512kB low:4352kB high:8192kB active_anon:105468kB inactive_anon:116604kB active_file:1584504kB inactive_file:1590276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:20kB mapped:6812kB shmem:40kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[163596.608634] lowmem_reserve[]: 0 0 0
[163596.608643] Normal: 216*4kB 215*8kB 216*16kB 216*32kB 36*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15256kB
[163596.608668] HighMem: 233*4kB 67*8kB 141*16kB 22*32kB 8*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5324kB
[163596.608692] 794329 total pagecache pages
[163596.608697] 12 pages in swap cache
[163596.608703] Swap cache stats: add 79, delete 67, find 9/11
[163596.608708] Free swap  = 8378092kB
[163596.608712] Total swap = 8378364kB
[163596.670667] 1046784 pages of RAM
[163596.670674] 6801 free pages
[163596.670679] 12533 reserved pages
[163596.670683] 36489 slab pages
[163596.670687] 631668 pages shared
[163596.670692] 12 pages swap cached
[163596.670701] SLUB: Unable to allocate memory on node -1 (gfp=0x8020)
[163596.670710]   cache: kmalloc-192, object size: 192, buffer size: 192, default order: 0, min order: 0
[163596.670718]   node 0: slabs: 2733, objs: 57393, free: 0

df -h:
...
/dev/sda6 216G 53G 163G 25% /mnt/sdb1

df -i:
...
/dev/sda6 56451072 19721920 36729152 35% /mnt/sdb1

flag@c13:~$ cat /proc/meminfo 
MemTotal:        4137004 kB
MemFree:         1191096 kB
Buffers:           23172 kB
Cached:          2074116 kB
SwapCached:           48 kB
Active:          1301568 kB
Inactive:        1016024 kB
Active(anon):     103748 kB
Inactive(anon):   116600 kB
Active(file):    1197820 kB
Inactive(file):   899424 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       3408896 kB
HighFree:        1108444 kB
LowTotal:         728108 kB
LowFree:           82652 kB
SwapTotal:       8378364 kB
SwapFree:        8378092 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        220284 kB
Mapped:             7068 kB
Shmem:                44 kB
Slab:             263216 kB
SReclaimable:     232212 kB
SUnreclaim:        31004 kB
KernelStack:        2192 kB
PageTables:         2312 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10446864 kB
Committed_AS:    1051868 kB
VmallocTotal:     245760 kB
VmallocUsed:        2360 kB
VmallocChunk:     241428 kB
flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6
   from      to extents  blocks    pct
      1       1   27058   27058   0.06
      2       3  124367  358831   0.84
      4       7   17656  121693   0.29
      8      15 2856900 42122381  98.81
total free extents 3025981
total free blocks 42629963
average free extent size 14.088
Signature · 7 lines
bye,
p.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw
Quoted 8 lines · Paolo Pisati
Quoted 3 lines · Dave Chinner

And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....

ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:

i'm testing a 3.5.y kernel plus those 3 patches:

549142a xfs: don't use speculative prealloc for small files f0843f4 xfs: limit speculative prealloc size on sparse files 454da09 xfs: inode allocation should use unmapped buffers.

and i can confirm that:

-using a small fs (2G) i cannot reproduce any -ENOSPC or vmalloc() problem anymore, the benchmark runs until running out of inodes

-using a bigger fs (~250G), two days and my tests are still running good

Signature · 7 lines
bye,
p.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
Quoted 16 lines · Paolo Pisati
Quoted 8 lines · Paolo Pisati
Quoted 3 lines · Dave Chinner

And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....

ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:

i'm testing a 3.5.y kernel plus those 3 patches:

549142a xfs: don't use speculative prealloc for small files f0843f4 xfs: limit speculative prealloc size on sparse files 454da09 xfs: inode allocation should use unmapped buffers.

Quoted 6 lines

and i can confirm that:

-using a small fs (2G) i cannot reproduce any -ENOSPC or vmalloc() problem anymore, the benchmark runs until running out of inodes

-using a bigger fs (~250G), two days and my tests are still running good

Ok, good to know. The first patch you list there hasn't even been reviewed yet, so it might take some time before that is ready for -stable backport.

Also, there are a bunch of fixes needed to the second patch you have there (f0843f4 xfs: limit speculative prealloc...) that would also be necessary for a -stable backport. i.e:

e8108ce xfs: fix xfs_iomap_eof_prealloc_initial_size type e114b5f xfs: increase prealloc size to double that of the previous extent e78c420 xfs: fix potential infinite loop in xfs_iomap_prealloc_size()

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
In reply to Paolo Pisati
Quoted 11 lines · Paolo Pisati
Quoted 3 lines · Dave Chinner

And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem....

ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this:

dmesg:
...

Quoted 13 lines
[163596.605253] updatedb.mlocat: page allocation failure: order:0, mode:0x20
[163596.605299] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04edb20>] (dump_stack+0x20/0x24)
[163596.605320] [<c04edb20>] (dump_stack+0x20/0x24) from [<c00e7780>] (warn_alloc_failed+0xd8/0x118)
[163596.605335] [<c00e7780>] (warn_alloc_failed+0xd8/0x118) from [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708)
[163596.605354] [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) from [<c011b798>] (new_slab+0x22c/0x248)
[163596.605370] [<c011b798>] (new_slab+0x22c/0x248) from [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8)
[163596.605383] [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) from [<c011ced4>] (kmem_cache_alloc+0x158/0x190)
[163596.605402] [<c011ced4>] (kmem_cache_alloc+0x158/0x190) from [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74)
[163596.605417] [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) from [<c0332c80>] (scsi_host_alloc_command+0x24/0x78)
[163596.605428] [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) from [<c0332cf0>] (__scsi_get_command+0x1c/0xa0)
[163596.605439] [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) from [<c0332db0>] (scsi_get_command+0x3c/0xb0)
[163596.605453] [<c0332db0>] (scsi_get_command+0x3c/0xb0) from [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60)
[163596.605466] [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) from [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac)

ENOMEM deep in the SCSI stack for an order 0 GFP_ATOMIC allocation. That's not an XFS problem - that's a SCSI stack issue. You should probably report that to the scsi list...

Quoted 11 lines
[163596.608574] active_anon:26367 inactive_anon:29153 isolated_anon:0
[163596.608574]  active_file:396338 inactive_file:397959 isolated_file:0
[163596.608574]  unevictable:0 dirty:0 writeback:5 unstable:0
[163596.608574]  free:5145 slab_reclaimable:57625 slab_unreclaimable:7729
[163596.608574]  mapped:1703 shmem:10 pagetables:581 bounce:0
[163596.608602] Normal free:15256kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:8kB active_file:848kB inactive_file:1560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:230500kB slab_unreclaimable:30916kB kernel_stack:2208kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[163596.608607] lowmem_reserve[]: 0 26423 26423
[163596.608628] HighMem free:5324kB min:512kB low:4352kB high:8192kB active_anon:105468kB inactive_anon:116604kB active_file:1584504kB inactive_file:1590276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:20kB mapped:6812kB shmem:40kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[163596.608634] lowmem_reserve[]: 0 0 0
[163596.608643] Normal: 216*4kB 215*8kB 216*16kB 216*32kB 36*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15256kB
[163596.608668] HighMem: 233*4kB 67*8kB 141*16kB 22*32kB 8*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5324kB

Though this says there is plenty of free order 0 pages in both low and high memory....

Quoted 6 lines
[163596.608692] 794329 total pagecache pages
[163596.608697] 12 pages in swap cache
[163596.608703] Swap cache stats: add 79, delete 67, find 9/11
[163596.608708] Free swap  = 8378092kB
[163596.608712] Total swap = 8378364kB
[163596.670667] 1046784 pages of RAM
Quoted 8 lines
[163596.670674] 6801 free pages
[163596.670679] 12533 reserved pages
[163596.670683] 36489 slab pages
[163596.670687] 631668 pages shared
[163596.670692] 12 pages swap cached
[163596.670701] SLUB: Unable to allocate memory on node -1 (gfp=0x8020)
[163596.670710]   cache: kmalloc-192, object size: 192, buffer size: 192, default order: 0, min order: 0
[163596.670718]   node 0: slabs: 2733, objs: 57393, free: 0

And it was slub that was unable to find a page when it should have been able to, so perhaps this is a VM problem?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs