Why does xfsdump encounter "WARNING: could not get list of non-root attributes for nondir ino nnnn: Cannot allocate memory (12)"?

Greetings,

Happy old year! ;-)

Running XFS on Ununtu 10.04 LTS, I have noticed more warnings with XFS dump the longer the system has been up. A reboot cuts down on the number of warnings xfsdump encounters. An example run of xfsdump is as follows:

/usr/sbin/xfsdump: using file dump (drive_simple) strategy
/usr/sbin/xfsdump: version 3.0.4 (dump format 3.0) - Running single-threaded
/usr/sbin/xfsdump: level 0 dump of ldslnx01:/srv
/usr/sbin/xfsdump: dump date: Sat Dec 31 15:25:42 2011
/usr/sbin/xfsdump: session id: b56b3cd7-30d6-44b4-a8a4-e131ee70b60d
/usr/sbin/xfsdump: session label: "data"
/usr/sbin/xfsdump: ino map phase 1: constructing initial dump list
/usr/sbin/xfsdump: ino map phase 2: skipping (no pruning necessary)
/usr/sbin/xfsdump: ino map phase 3: skipping (only one dump stream)
/usr/sbin/xfsdump: ino map construction complete
/usr/sbin/xfsdump: estimated dump size: 115786343168 bytes
/usr/sbin/xfsdump: WARNING: no media label specified
/usr/sbin/xfsdump: creating dump session media file 0 (media 0, file 0)
/usr/sbin/xfsdump: dumping ino map
/usr/sbin/xfsdump: dumping directories
/usr/sbin/xfsdump: dumping non-directory files
/usr/sbin/xfsdump: WARNING: could not get list of non-root attributes for nondir ino 234778: Cannot allocate memory (12) /usr/sbin/xfsdump: WARNING: could not get list of non-root attributes for nondir ino 234813: Cannot allocate memory (12) /usr/sbin/xfsdump: WARNING: could not get list of non-root attributes for nondir ino 1828443: Cannot allocate memory (12) /usr/sbin/xfsdump: WARNING: could not get list of root attributes for nondir ino 1828443: Cannot allocate memory (12) /usr/sbin/xfsdump: WARNING: could not get list of non-root attributes for nondir ino 2234019: Cannot allocate memory (12) /usr/sbin/xfsdump: ending media file
/usr/sbin/xfsdump: media file size 115563111376 bytes
/usr/sbin/xfsdump: dump size (non-dir files) : 115472209824 bytes
/usr/sbin/xfsdump: dump complete: 4099 seconds elapsed
/usr/sbin/xfsdump: Dump Status: SUCCESS

Could someone please explain what is going on, and why a freshly rebooted system would result in less / no errors?

Sincerely,

Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

It's getting ENOMEM from the kernel.

xfsdump is asking for the list of attributes from the file, and passing in a buffer of XATTR_LIST_MAX bytes, which the xfs_attrmulti_attr_get() function immeidately kmalloc()s for a working buffer. That kmalloc is likely to be failing randomly as it is asking for a contiguous 64k allocation, and that may not be available given memory gets fragmented over time.

Memory is less fragmented, hence more likely to have a order-5 allocation succeed. Try the patch below, and see if that helps.

Cheers,

Dave.

Dave Chinner
[email protected]


xfs: handle kmalloc failure when reading attrs

From: Dave Chinner <[email protected]>

xfsdump uses for a large buffer for extended attributes, which has a
kmalloc'd shadow buffer in the kernel. This can fail after the
system has been running for some time as it is a high order
allocation. Convert this to a vmalloc so that it doesn't require
contiguous memory and so won't randomly fail while xfsdump is
running.

Signed-off-by: Dave Chinner <[email protected]>
---
 fs/xfs/xfs_ioctl.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

index 2f3f56a..0f3fab1 100644
@@ -448,7 +448,7 @@ xfs_attrmulti_attr_get(
 
 	if (*len > XATTR_SIZE_MAX)
 		return EINVAL;
-	kbuf = kmalloc(*len, GFP_KERNEL);
+	kbuf = vmalloc(*len, GFP_KERNEL);
 	if (!kbuf)
 		return ENOMEM;
 

@@ -460,7 +460,7 @@ xfs_attrmulti_attr_get(
 		error = EFAULT;
 
  out_kfree:
-	kfree(kbuf);
+	vfree(kbuf);
 	return error;
 }
 

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

More I was feeling a bit guilty that I was bouncing the machines to have the backup work without error... as "Linux Administrators NEVER reboot their boxes!"

I have a few too many things in the test tubes at the moment to take on custom patching a kernel build.

So, do these mean that there is anything wrong with the backup being made? Is the final SUCCESS return code in error?

Thank you for providing an explanation and patch.

Sincerely,

Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

I think this should use kmalloc for allocation smaller than a page to avoid the vmalloc overhead for the usual case of small attributes.

Also the fs/xattr.c code for the normal xattr system calls needs the same treatment.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Sounds like a generic kvmalloc()/kvfree() call pair is the answer then. i.e. if kmalloc fails, then use vmalloc()...

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

If my memory serves me right that's been discussed and rejected a few times before.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Which has lead to several implementations of duplicated functionality like:

- ext4_kvmalloc/ext4_kvfree
- drm_malloc_ab/drm_calloc_large/drm_free_large
- ceph_buffer_new/ceph_buffer_release
- gfs2_alloc_sort_buffer/gfs2_free_sort_buffer
- __ntfs_malloc/ntfs_free

I don't think adding another two sets of identical wrappers (or open coding it) is the right thing given that it is a repeating pattern.

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Give it a try, I'm not against it myself.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs