xfs_bmap Cannot allocate memory
Hi List!
I've got a file here on which i cannot use xfs_bmap to determine it's fragments. All that i know is that it must have a really great number of them. It was the result of running a smbd without strict allocate. The machine itself has 8GiB of RAM and 10GiB of swap available, so that shouldn't be the problem. I guess this is some bug in xfs_bmap. Or is it a known limitation?
# xfs_bmap /backup/tmp/cannot_allocate_memory.vhd xfs_bmap: xfsctl(XFS_IOC_GETBMAPX) iflags=0x0 ["/backup/tmp/cannot_allocate_memory.vhd"]: Cannot allocate memory
Another thing i noticed is that when i use the filefrag utility it reports a different fragment count when invoked with -v than without it, which i found pretty strange too. From that output i judge that the real count is 62715, which is also not in sync with 45487.
# filefrag /backup/tmp/cannot_allocate_memory.vhd /backup/tmp/cannot_allocate_memory.vhd: 1364 extents found # filefrag -v /backup/tmp/cannot_allocate_memory.vhd | tail -n5 62711 43265805 57280748 57265654 17 62712 43266061 57265655 57280764 1 62713 43266317 57396971 57265655 17 62714 43266573 57265656 57396987 1 eof /backup/tmp/cannot_allocate_memory.vhd: 45487 extents found
Other than that everything works perfectly well and xfs_repair doesn't report any problems with the filesystem. So it's only a "cosmetical" issue.
Thanks for any hints,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Michael Weissenbacher
Hi List!
I've got a file here on which i cannot usexfs_bmapto determine it's fragments. All that i know is that it must have a really great number of them. It was the result of running a smbd without strict allocate. The machine itself has 8GiB of RAM and 10GiB of swap available, so that shouldn't be the problem. I guess this is some bug inxfs_bmap. Or is it a known limitation?
Sounds like your memory is fragmented. IIRC, bmap tries to map all the extents in a single buffer, and that might cause problems for files with large numbers of extents. ENOMEM can occur if an internal buffer cannot be allocated to hold all the extents to be mapped in one call.
Try using the "-n <num_extents>" option to reduce the number of extents gathered per ioctl call and see if that makes the issue go away.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave!
Quoted 11 lines
Sounds like your memory is fragmented. IIRC, bmap tries to map all the extents in a single buffer, and that might cause problems for files with large numbers of extents. ENOMEM can occur if an internal buffer cannot be allocated to hold all the extents to be mapped in one call.
Try using the "-n <
num_extents>" option to reduce the number of extents gathered per ioctl call and see if that makes the issue go away.
Thanks, i've tried that:
# xfs_bmap -n 3 /backup/tmp/cannot_allocate_memory.vhd /backup/tmp/cannot_allocate_memory.vhd: 0: [0..134279]: 444610560..444744839 1: [134280..134399]: hole 2: [134400..206495]: 433472688..433544783 # xfs_bmap -n 70000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 69999: [244690864..244690871]: 1173913592..1173913599 # xfs_bmap -n 75000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 74999: [253425664..253425671]: 1284986768..1284986775 # xfs_bmap -n 80000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 79999: [262287488..262289015]: hole # xfs_bmap -n 85000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 84999: [272607184..272613335]: 1497107288..1497113439 # xfs_bmap -n 90000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 xfs_bmap: xfsctl(XFS_IOC_GETBMAPX) iflags=0x0 ["/backup/tmp/cannot_allocate_memory.vhd"]: Cannot allocate memory
- Seems that xfs_bmap reads at maximum the number of extents that i specified with -n
- Seems that the file has even more then 85000 extents
xfsprogs version is 3.1.5, compiled from source
cheers,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 30 lines · Michael Weissenbacher
Hi Dave!
Quoted 11 lines
Sounds like your memory is fragmented. IIRC, bmap tries to map all the extents in a single buffer, and that might cause problems for files with large numbers of extents. ENOMEM can occur if an internal buffer cannot be allocated to hold all the extents to be mapped in one call.
Try using the "-n <
num_extents>" option to reduce the number of extents gathered per ioctl call and see if that makes the issue go away.Thanks, i've tried that:
# xfs_bmap -n 3 /backup/tmp/cannot_allocate_memory.vhd /backup/tmp/cannot_allocate_memory.vhd: 0: [0..134279]: 444610560..444744839 1: [134280..134399]: hole 2: [134400..206495]: 433472688..433544783 # xfs_bmap -n 70000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 69999: [244690864..244690871]: 1173913592..1173913599 # xfs_bmap -n 75000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 74999: [253425664..253425671]: 1284986768..1284986775 # xfs_bmap -n 80000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 79999: [262287488..262289015]: hole # xfs_bmap -n 85000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 84999: [272607184..272613335]: 1497107288..1497113439 # xfs_bmap -n 90000 /backup/tmp/cannot_allocate_memory.vhd | tail -n1 xfs_bmap: xfsctl(XFS_IOC_GETBMAPX) iflags=0x0 ["/backup/tmp/cannot_allocate_memory.vhd"]: Cannot allocate memory
Quoted 3 lines
- Seems that
xfs_bmapreads at maximum the number of extents that i specified with -n
- Seems that the file has even more then 85000 extents
Ah, there's a mismatch betwenteh man page and the implementation, then. The man page implies that "-n <num>" means query num extents at a time to map the entire file. It's implemented as "map the first <num> extents", though.
You could try this:
# xfs_io -f -c "fiemap -v" <file>
Because fiemap loops doing getting a small number of extents at a time...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave!
Quoted 6 lines
Ah, there's a mismatch betwenteh man page and the implementation, then. The man page implies that "-n <num>" means query num extents at a time to map the entire file. It's implemented as "map the first <num> extents", though.
Quoted 7 lines
You could try this:
# xfs_io -f -c "fiemap -v" <file>Because fiemap loops doing getting a small number of extents at a time...
Ok, after i found out that fiemap is bleeding-edge (xfsprogs 3.1.5 doesn't support it) i compiled the latest git version of xfsprogs.
"fiemap -v" doesn't bail out on this file but continues to print extents forever. I stopped it after the redirected output had already reached 90GiB and printed negative numbers. I guess it must have run into an endless loop somewhere. xfs_bmap from the git version also throws "Cannot allocate memory" at me.
# ./io/xfs_io -f -c "fiemap" /backup/tmp/cannot_allocate_memory.vhd > /tmp/fiemap_output.txt (Control-C)
# ls -lh /tmp/fiemap_output.txt -rw-r--r-- 1 root root 90G 2011-07-05 15:06 /tmp/fiemap_output.txt
# tail -n10 /tmp/fiemap_output.txt -1925931740: [599928..602239]: 399016..401327 -1925931739: [602240..603895]: hole -1925931738: [603896..603903]: 2480..2487 -1925931737: [603904..606199]: hole -1925931736: [606200..608127]: 366632..368559 -1925931735: [608128..1183]: hole -1925931734: [1184..134279]: 444611744..444744839 -1925931733: [134280..134399]: hole -1925931732: [134400..206495]: 433472688..433544783 -1925931731: [206496..210455]: hole
# ./io/xfs_io -f -c "bmap" /backup/tmp/cannot_allocate_memory.vhd xfs_io: xfsctl(XFS_IOC_GETBMAPX) iflags=0x0 ["/backup/tmp/cannot_allocate_memory.vhd"]: Cannot allocate memory
cheers,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 1 line · Michael Weissenbacher
xfs_bmap:xfsctl(XFS_IOC_GETBMAPX)iflags=0x0
Googling the text above returns only 5 results, including your post today to this list. Thus I doubt your problem is the result of a bug in XFS or xfsprogs.
Was the file being written to when you ran xfs_bmap?
If not, did you flush caches before running xfs_bmap? Did you try unmounting and remounting the filesystem? These steps are sometimes needed to get actual extent usage due to extents in cache that have not yet been written to disk.
What distro/kernel version?
What xfsprogs version?
What filefrag version?
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Stan Hoeppner
Quoted 1 line · Michael Weissenbacher
xfs_bmap:xfsctl(XFS_IOC_GETBMAPX)iflags=0x0Googling the text above returns only 5 results, including your post today to this list. Thus I doubt your problem is the result of a bug in XFS or xfsprogs.
Was the file being written to when you ran
xfs_bmap?
No, the file has last been written 4 days ago
Quoted 4 lines
If not, did you flush caches before running
xfs_bmap? Did you try unmounting and remounting the filesystem? These steps are sometimes needed to get actual extent usage due to extents in cache that have not yet been written to disk.
Yes, i tried flushing the caches "echo 3 >/proc/sys/vm/drop_caches" which didn't change anything.
I also did unmount, xfs_check (no errors reported), mount which didn't change anything.
Quoted 1 line
What distro/kernel version?
Ubuntu Lucid 10.04.2 LTS with vanilla kernel 2.6.39.2
Quoted 1 line
What xfsprogs version?
Latest version 3.1.5, compiled from sources
Quoted 1 line
What filefrag version?
e2fsprogs 1.41.11-1ubuntu2.1
cheers,
Michael
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Michael Weissenbacher
Hi List!
I've got a file here on which i cannot usexfs_bmapto determine it's fragments. All that i know is that it must have a really great number of them. It was the result of running a smbd without strict allocate. The machine itself has 8GiB of RAM and 10GiB of swap available, so that shouldn't be the problem. I guess this is some bug inxfs_bmap. Or is it a known limitation?# xfs_bmap /backup/tmp/cannot_allocate_memory.vhd xfs_bmap: xfsctl(XFS_IOC_GETBMAPX) iflags=0x0 ["/backup/tmp/cannot_allocate_memory.vhd"]: Cannot allocate memory
What version of xfsprogs do you use? xfsprogs 3.0.1 fixes a problem in the bmap command that looked pretty similar to what you see.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hi Christoph!
Quoted 5 lines
What version of xfsprogs do you use? xfsprogs 3.0.1 fixes a problem in the bmap command that looked pretty similar to what you see.
Sorry, i forgot to mention that in my original post:
# xfs_bmap -V xfs_bmap version 3.1.5
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs