NTFS - Kernel memory leak in driver for kernel 2.4.28?
I am maintaining an embedded linux environment that is used for disaster data recovery for Xpoint Technologies. The environment is based on kernel 2.4.28, and a 32megabyte ramdisk containing bash, a minimal busybox environment, and the backup software with no swap.
During software development to add a new feature to restore individual files from an NTFS partition, the software started to exhibit out of memory errors.
The following procedure can be used to replicate the error :
Boot a 2.4.28 kernel, with a 32meg ramdisk and minimal busybox environment in the initrd ramdisk with the "find" utility. The NTFS driver can be compiled in, or can be a module. This has also been experimentally verified using a small program that replicates "find"
with careful attention to malloc( ) and free ( ). However, the
standard "find" utility suffices to show the problem :
Note the available memory in /proc/meminfo, and look at /proc/slabinfo
Mount a large NTFS filesystem, and then run : find /path/to/ntfs -name *
Note the available memory in /proc/meminfo, and look at /proc/slabinfo
Unmounting the ntfs filesystem, and unloading the kernel module (when compiled as such) had no effect on the amount of free ram available
after the 'find' command is run. malloc( ) fails to allocate
afterwards. so the kernel does not free any of the missing RAM for malloc( ).
-------------------------------------------------------------------------
On a testbed machine (P4, 256Mb RAM = IBM Netvista) :
Free Memory before : 209768448 (~200 Mb)
Free Memory after : 3579904 (~35 Mb)
Linux (none) 2.4.28 #1 Tue Feb 15 10:25:25 EST 2005 i686 unknown
(/proc/meminfo before)
total: used: free: shared: buffers: cached:
Mem: 252907520 43139072 209768448 0 385024 38862848 Swap: 0 0 0 MemTotal: 246980 kB MemFree: 204852 kB MemShared: 0 kB Buffers: 376 kB Cached: 37952 kB SwapCached: 0 kB Active: 2784 kB Inactive: 35552 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 246980 kB LowFree: 204852 kB SwapTotal: 0 kB SwapFree: 0 kB
(/proc/slabinfo before)
slabinfo - version: 1.1
kmem_cache 61 72 108 2 2 1 tcp_tw_bucket 0 0 96 0 0 1 tcp_bind_bucket 0 0 32 0 0 1 tcp_open_request 0 0 64 0 0 1 inet_peer_cache 0 0 64 0 0 1 ip_fib_hash 4 113 32 1 1 1 ip_dst_cache 0 0 160 0 0 1 arp_cache 0 0 96 0 0 1 blkdev_requests 3072 4120 96 77 103 1 nfs_write_data 0 0 352 0 0 1 nfs_read_data 0 0 352 0 0 1 nfs_page 0 0 96 0 0 1 journal_head 0 0 48 0 0 1 revoke_table 0 0 12 0 0 1 revoke_record 0 0 32 0 0 1 dnotify_cache 0 0 20 0 0 1 file_lock_cache 0 0 92 0 0 1 fasync_cache 0 0 16 0 0 1 uid_cache 0 0 32 0 0 1 skbuff_head_cache 1 24 160 1 1 1 sock 5 9 864 1 1 2 sigqueue 0 29 132 0 1 1 kiobuf 0 0 64 0 0 1 cdev_cache 6 59 64 1 1 1 bdev_cache 3 59 64 1 1 1 mnt_cache 11 59 64 1 1 1 inode_cache 167 168 480 21 21 1 dentry_cache 224 240 128 8 8 1 filp 37 60 128 2 2 1 names_cache 0 2 4096 0 2 1 buffer_head 5941 5960 96 149 149 1 mm_struct 7 24 160 1 1 1 vm_area_struct 69 120 96 2 3 1 fs_cache 6 113 32 1 1 1 files_cache 7 18 416 1 2 1 signal_act 8 12 1312 3 4 1 size-131072(DMA) 0 0 131072 0 0 32 size-131072 0 0 131072 0 0 32 size-65536(DMA) 0 0 65536 0 0 16 size-65536 0 0 65536 0 0 16 size-32768(DMA) 0 0 32768 0 0 8 size-32768 0 1 32768 0 1 8 size-16384(DMA) 0 0 16384 0 0 4 size-16384 1 2 16384 1 2 4 size-8192(DMA) 0 0 8192 0 0 2 size-8192 4 5 8192 4 5 2 size-4096(DMA) 0 0 4096 0 0 1 size-4096 6 6 4096 6 6 1 size-2048(DMA) 0 0 2048 0 0 1 size-2048 5 6 2048 3 3 1 size-1024(DMA) 0 4 1024 0 1 1 size-1024 95 96 1024 24 24 1 size-512(DMA) 0 0 512 0 0 1 size-512 28 32 512 4 4 1 size-256(DMA) 0 0 256 0 0 1 size-256 6 15 256 1 1 1 size-128(DMA) 0 0 128 0 0 1 size-128 490 510 128 17 17 1 size-64(DMA) 0 0 64 0 0 1 size-64 108 118 64 2 2 1 size-32(DMA) 0 0 32 0 0 1 size-32 206 339 32 2 3 1
----------- run : find /mnt/ntfs -name \*
(/proc/meminfo after)
total: used: free: shared: buffers: cached:
Mem: 252907520 249327616 3579904 0 39251968 37748736 Swap: 0 0 0 MemTotal: 246980 kB MemFree: 3496 kB MemShared: 0 kB Buffers: 38332 kB Cached: 36864 kB SwapCached: 0 kB Active: 67084 kB Inactive: 8120 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 246980 kB LowFree: 3496 kB SwapTotal: 0 kB SwapFree: 0 kB
(/proc/slabinfo after)
slabinfo - version: 1.1
kmem_cache 61 72 108 2 2 1 tcp_tw_bucket 0 0 96 0 0 1 tcp_bind_bucket 0 0 32 0 0 1 tcp_open_request 0 0 64 0 0 1 inet_peer_cache 0 0 64 0 0 1 ip_fib_hash 4 113 32 1 1 1 ip_dst_cache 0 0 160 0 0 1 arp_cache 0 0 96 0 0 1 blkdev_requests 3072 3600 96 77 90 1 nfs_write_data 0 0 352 0 0 1 nfs_read_data 0 0 352 0 0 1 nfs_page 0 0 96 0 0 1 journal_head 0 0 48 0 0 1 revoke_table 0 0 12 0 0 1 revoke_record 0 0 32 0 0 1 dnotify_cache 0 0 20 0 0 1 file_lock_cache 0 0 92 0 0 1 fasync_cache 0 0 16 0 0 1 uid_cache 0 0 32 0 0 1 skbuff_head_cache 1 24 160 1 1 1 sock 5 9 864 1 1 2 sigqueue 0 29 132 0 1 1 kiobuf 0 0 64 0 0 1 cdev_cache 5 59 64 1 1 1 bdev_cache 3 59 64 1 1 1 mnt_cache 12 59 64 1 1 1 inode_cache 26560 26560 480 3320 3320 1 dentry_cache 20721 21840 128 728 728 1 filp 42 60 128 2 2 1 names_cache 0 1 4096 0 1 1 buffer_head 80852 80880 96 2022 2022 1 mm_struct 7 24 160 1 1 1 vm_area_struct 69 120 96 2 3 1 fs_cache 6 113 32 1 1 1 files_cache 7 18 416 1 2 1 signal_act 8 12 1312 3 4 1 size-131072(DMA) 0 0 131072 0 0 32 size-131072 0 0 131072 0 0 32 size-65536(DMA) 0 0 65536 0 0 16 size-65536 0 0 65536 0 0 16 size-32768(DMA) 0 0 32768 0 0 8 size-32768 0 1 32768 0 1 8 size-16384(DMA) 0 0 16384 0 0 4 size-16384 1 2 16384 1 2 4 size-8192(DMA) 0 0 8192 0 0 2 size-8192 4 5 8192 4 5 2 size-4096(DMA) 0 0 4096 0 0 1 size-4096 22252 22288 4096 22252 22288 1 size-2048(DMA) 0 0 2048 0 0 1 size-2048 5 6 2048 3 3 1 size-1024(DMA) 0 4 1024 0 1 1 size-1024 26781 26992 1024 6696 6748 1 size-512(DMA) 0 0 512 0 0 1 size-512 27801 27960 512 3476 3495 1 size-256(DMA) 0 0 256 0 0 1 size-256 1718 1725 256 115 115 1 size-128(DMA) 0 0 128 0 0 1 size-128 57680 58050 128 1923 1935 1 size-64(DMA) 0 0 64 0 0 1 size-64 12190 12213 64 207 207 1 size-32(DMA) 0 0 32 0 0 1 size-32 55206 55483 32 491 491 1
NTFS - Kernel memory leak in driver for kernel 2.4.28 (update)
I noticed that the inode_cache is quite high. No matter what I do to
the tunable system parameters in /proc (below) I can't get the amount of memory taken by the inode cache to reduce and become freed for userspace programs.
/proc/sys/vm/vm_mapped_ratio ( tried 20 - 100 )/proc/sys/vm/vm_vfs_scan_ratio ( tried 2 - 6 )
How often does the dcache/icache shrinking take place in 2.4?
On vfat/fat32, ext2/3 filesystems I don't see anywhere near the amount of lost memory than when using the NTFS filesystem. This is what led me to think that the NTFS driver was leaking, rather than this being an inode cache problem.
On Wed, 16 Feb 2005, Martin Bogomolni wrote:
[SNIPPED...]
Quoted 4 lines
after the 'find' command is run.
malloc( )fails to allocate
afterwards. so the kernel does not free any of the missing RAM formalloc( ).
Whatever program is using malloc() is either corrupting its buffers or has a memory leak.
Malloc() will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pages malloc() "promised".
When you look at kernel memory after using find everything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes.
So, the bottom line is, if malloc() returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting.
[SNIPPED...]
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
Dick,
I should say that the malloc() succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer.
After taking another look at the problem, the kernel has a lot of
memory tied up in the inode and dentry cache. I've tuned/proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio with no real success in shrinking the amount of memory used by these caches.
Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it?
-Martin
Quoted 34 lines · linux-os
On Wed, 16 Feb 2005, Martin Bogomolni wrote:
[SNIPPED...]
Quoted 4 lines
after the 'find' command is run.
malloc( )fails to allocate
afterwards. so the kernel does not free any of the missing RAM formalloc( ).Whatever program is using
malloc()is either corrupting its buffers or has a memory leak.
Malloc()will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pagesmalloc()"promised".When you look at kernel memory after using
findeverything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes.So, the bottom line is, if
malloc()returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting.[SNIPPED...]
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
kernel 2.4 inode/dentry cache not clearing on umount?
Also .. David :
Are you saying that, on a system with 256Megs of ram, of which the kernel is reporting only 3-4Mb free because the inode/dentry caches are taking up most of the memory, and NO page/swap file....
char *p;
p = (char *) malloc( 64*1024*1024 );
I assure you that under these conditions, the malloc( ) will fail with NULL.
---------------------------------
Now, in the meantime I have discovered that merely unmounting the filesystem is not enough to clear the dcache and icache.
However, if I unmount the filesystem then run:
cat /dev/hda > /dev/null
This causes the inode/dentry cache to finally shrink and the amount of
available free memory increases back to ~200Mb. However, this
reduction does not immediately take place when the filesystem is unmounted, and while the filesystem is mounted .. the inode/dentry cache does not shrink and leaves only 3Mb of available free memory.
Quoted 57 lines · Martin Bogomolni
Dick,
I should say that the
malloc()succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer.After taking another look at the problem, the kernel has a lot of
memory tied up in the inode and dentry cache. I've tuned/proc/sys/vm/vm_cache_scan_ratio,vm_mapped_ratio,vm_vfs_scan_ratiowith no real success in shrinking the amount of memory used by these caches.Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it?
-Martin
Quoted 34 lines · linux-os
On Wed, 16 Feb 2005, Martin Bogomolni wrote:
[SNIPPED...]
Quoted 4 lines
after the 'find' command is run.
malloc( )fails to allocate
afterwards. so the kernel does not free any of the missing RAM formalloc( ).Whatever program is using
malloc()is either corrupting its buffers or has a memory leak.
Malloc()will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pagesmalloc()"promised".When you look at kernel memory after using
findeverything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes.So, the bottom line is, if
malloc()returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting.[SNIPPED...]
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
kernel 2.4 inode/dentry cache not clearing on umount?
Also .. David :
Are you saying that, on a system with 256Megs of ram, of which the kernel is reporting only 3-4Mb free because the inode/dentry caches are taking up most of the memory, and NO page/swap file....
char *p;
p = (char *) malloc( 64*1024*1024 );
I assure you that under these conditions, the malloc( ) will fail with NULL.
---------------------------------
Now, in the meantime I have discovered that merely unmounting the filesystem is not enough to clear the dcache and icache.
However, if I unmount the filesystem then run:
cat /dev/hda > /dev/null
This causes the inode/dentry cache to finally shrink and the amount of
available free memory increases back to ~200Mb. However, this
reduction does not immediately take place when the filesystem is unmounted, and while the filesystem is mounted .. the inode/dentry cache does not shrink and leaves only 3Mb of available free memory.
Quoted 11 lines · Martin Bogomolni
I should say that the
malloc()succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer.After taking another look at the problem, the kernel has a lot of
memory tied up in the inode and dentry cache. I've tuned/proc/sys/vm/vm_cache_scan_ratio,vm_mapped_ratio,vm_vfs_scan_ratiowith no real success in shrinking the amount of memory used by these caches.
Quoted 3 lines
Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it?
Are you using the old (1.x) or new (2.x) ntfs driver?
If you are using the old one you could try using the new driver. That should be a lot better than the old one in terms of how much memory it uses. You can get an outdated patch (but should still work) for the new driver here:
http://sourceforge.net/project/showfiles.php?group_id=13956&package_id=21892
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/