Skip to contentSkip to conversation

NTFS - Kernel memory leak in driver for kernel 2.4.28?

DISCUSSION Martin Bogomolni 3 participants 7 messages Feb 16, 2005 – Feb 17, 2005

lore
Share & tools
lore
#0 lore raw

I am maintaining an embedded linux environment that is used for disaster data recovery for Xpoint Technologies. The environment is based on kernel 2.4.28, and a 32megabyte ramdisk containing bash, a minimal busybox environment, and the backup software with no swap.

During software development to add a new feature to restore individual files from an NTFS partition, the software started to exhibit out of memory errors.

The following procedure can be used to replicate the error :

Boot a 2.4.28 kernel, with a 32meg ramdisk and minimal busybox environment in the initrd ramdisk with the "find" utility. The NTFS driver can be compiled in, or can be a module. This has also been experimentally verified using a small program that replicates "find"
with careful attention to malloc( ) and free ( ). However, the
standard "find" utility suffices to show the problem :

Note the available memory in /proc/meminfo, and look at /proc/slabinfo

Mount a large NTFS filesystem, and then run : find /path/to/ntfs -name *

Note the available memory in /proc/meminfo, and look at /proc/slabinfo

Unmounting the ntfs filesystem, and unloading the kernel module (when compiled as such) had no effect on the amount of free ram available
after the 'find' command is run. malloc( ) fails to allocate
afterwards. so the kernel does not free any of the missing RAM for malloc( ).

-------------------------------------------------------------------------

On a testbed machine (P4, 256Mb RAM = IBM Netvista) :

Free Memory before : 209768448 (~200 Mb)
Free Memory after : 3579904 (~35 Mb)

Linux (none) 2.4.28 #1 Tue Feb 15 10:25:25 EST 2005 i686 unknown

(/proc/meminfo before)

total: used: free: shared: buffers: cached:

Mem:  252907520 43139072 209768448        0   385024 38862848
Swap:        0        0        0
MemTotal:       246980 kB
MemFree:        204852 kB
MemShared:           0 kB
Buffers:           376 kB
Cached:          37952 kB
SwapCached:          0 kB
Active:           2784 kB
Inactive:        35552 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       246980 kB
LowFree:        204852 kB
SwapTotal:           0 kB
SwapFree:            0 kB

(/proc/slabinfo before)

slabinfo - version: 1.1

kmem_cache            61     72    108    2    2    1
tcp_tw_bucket          0      0     96    0    0    1
tcp_bind_bucket        0      0     32    0    0    1
tcp_open_request       0      0     64    0    0    1
inet_peer_cache        0      0     64    0    0    1
ip_fib_hash            4    113     32    1    1    1
ip_dst_cache           0      0    160    0    0    1
arp_cache              0      0     96    0    0    1
blkdev_requests     3072   4120     96   77  103    1
nfs_write_data         0      0    352    0    0    1
nfs_read_data          0      0    352    0    0    1
nfs_page               0      0     96    0    0    1
journal_head           0      0     48    0    0    1
revoke_table           0      0     12    0    0    1
revoke_record          0      0     32    0    0    1
dnotify_cache          0      0     20    0    0    1
file_lock_cache        0      0     92    0    0    1
fasync_cache           0      0     16    0    0    1
uid_cache              0      0     32    0    0    1
skbuff_head_cache      1     24    160    1    1    1
sock                   5      9    864    1    1    2
sigqueue               0     29    132    0    1    1
kiobuf                 0      0     64    0    0    1
cdev_cache             6     59     64    1    1    1
bdev_cache             3     59     64    1    1    1
mnt_cache             11     59     64    1    1    1
inode_cache          167    168    480   21   21    1
dentry_cache         224    240    128    8    8    1
filp                  37     60    128    2    2    1
names_cache            0      2   4096    0    2    1
buffer_head         5941   5960     96  149  149    1
mm_struct              7     24    160    1    1    1
vm_area_struct        69    120     96    2    3    1
fs_cache               6    113     32    1    1    1
files_cache            7     18    416    1    2    1
signal_act             8     12   1312    3    4    1
size-131072(DMA)       0      0 131072    0    0   32
size-131072            0      0 131072    0    0   32
size-65536(DMA)        0      0  65536    0    0   16
size-65536             0      0  65536    0    0   16
size-32768(DMA)        0      0  32768    0    0    8
size-32768             0      1  32768    0    1    8
size-16384(DMA)        0      0  16384    0    0    4
size-16384             1      2  16384    1    2    4
size-8192(DMA)         0      0   8192    0    0    2
size-8192              4      5   8192    4    5    2
size-4096(DMA)         0      0   4096    0    0    1
size-4096              6      6   4096    6    6    1
size-2048(DMA)         0      0   2048    0    0    1
size-2048              5      6   2048    3    3    1
size-1024(DMA)         0      4   1024    0    1    1
size-1024             95     96   1024   24   24    1
size-512(DMA)          0      0    512    0    0    1
size-512              28     32    512    4    4    1
size-256(DMA)          0      0    256    0    0    1
size-256               6     15    256    1    1    1
size-128(DMA)          0      0    128    0    0    1
size-128             490    510    128   17   17    1
size-64(DMA)           0      0     64    0    0    1
size-64              108    118     64    2    2    1
size-32(DMA)           0      0     32    0    0    1
size-32              206    339     32    2    3    1

----------- run : find /mnt/ntfs -name \*

(/proc/meminfo after)

total: used: free: shared: buffers: cached:

Mem:  252907520 249327616  3579904        0 39251968 37748736
Swap:        0        0        0
MemTotal:       246980 kB
MemFree:          3496 kB
MemShared:           0 kB
Buffers:         38332 kB
Cached:          36864 kB
SwapCached:          0 kB
Active:          67084 kB
Inactive:         8120 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       246980 kB
LowFree:          3496 kB
SwapTotal:           0 kB
SwapFree:            0 kB

(/proc/slabinfo after)

slabinfo - version: 1.1

kmem_cache            61     72    108    2    2    1
tcp_tw_bucket          0      0     96    0    0    1
tcp_bind_bucket        0      0     32    0    0    1
tcp_open_request       0      0     64    0    0    1
inet_peer_cache        0      0     64    0    0    1
ip_fib_hash            4    113     32    1    1    1
ip_dst_cache           0      0    160    0    0    1
arp_cache              0      0     96    0    0    1
blkdev_requests     3072   3600     96   77   90    1
nfs_write_data         0      0    352    0    0    1
nfs_read_data          0      0    352    0    0    1
nfs_page               0      0     96    0    0    1
journal_head           0      0     48    0    0    1
revoke_table           0      0     12    0    0    1
revoke_record          0      0     32    0    0    1
dnotify_cache          0      0     20    0    0    1
file_lock_cache        0      0     92    0    0    1
fasync_cache           0      0     16    0    0    1
uid_cache              0      0     32    0    0    1
skbuff_head_cache      1     24    160    1    1    1
sock                   5      9    864    1    1    2
sigqueue               0     29    132    0    1    1
kiobuf                 0      0     64    0    0    1
cdev_cache             5     59     64    1    1    1
bdev_cache             3     59     64    1    1    1
mnt_cache             12     59     64    1    1    1
inode_cache        26560  26560    480 3320 3320    1
dentry_cache       20721  21840    128  728  728    1
filp                  42     60    128    2    2    1
names_cache            0      1   4096    0    1    1
buffer_head        80852  80880     96 2022 2022    1
mm_struct              7     24    160    1    1    1
vm_area_struct        69    120     96    2    3    1
fs_cache               6    113     32    1    1    1
files_cache            7     18    416    1    2    1
signal_act             8     12   1312    3    4    1
size-131072(DMA)       0      0 131072    0    0   32
size-131072            0      0 131072    0    0   32
size-65536(DMA)        0      0  65536    0    0   16
size-65536             0      0  65536    0    0   16
size-32768(DMA)        0      0  32768    0    0    8
size-32768             0      1  32768    0    1    8
size-16384(DMA)        0      0  16384    0    0    4
size-16384             1      2  16384    1    2    4
size-8192(DMA)         0      0   8192    0    0    2
size-8192              4      5   8192    4    5    2
size-4096(DMA)         0      0   4096    0    0    1
size-4096          22252  22288   4096 22252 22288    1
size-2048(DMA)         0      0   2048    0    0    1
size-2048              5      6   2048    3    3    1
size-1024(DMA)         0      4   1024    0    1    1
size-1024          26781  26992   1024 6696 6748    1
size-512(DMA)          0      0    512    0    0    1
size-512           27801  27960    512 3476 3495    1
size-256(DMA)          0      0    256    0    0    1
size-256            1718   1725    256  115  115    1
size-128(DMA)          0      0    128    0    0    1
size-128           57680  58050    128 1923 1935    1
size-64(DMA)           0      0     64    0    0    1
size-64            12190  12213     64  207  207    1
size-32(DMA)           0      0     32    0    0    1
size-32            55206  55483     32  491  491    1
#1 lore raw

NTFS - Kernel memory leak in driver for kernel 2.4.28 (update)

I noticed that the inode_cache is quite high. No matter what I do to
the tunable system parameters in /proc (below) I can't get the amount of memory taken by the inode cache to reduce and become freed for userspace programs.

/proc/sys/vm/vm_mapped_ratio ( tried 20 - 100 )
/proc/sys/vm/vm_vfs_scan_ratio ( tried 2 - 6 )

How often does the dcache/icache shrinking take place in 2.4?

On vfat/fat32, ext2/3 filesystems I don't see anywhere near the amount of lost memory than when using the NTFS filesystem. This is what led me to think that the NTFS driver was leaking, rather than this being an inode cache problem.

#2 lore raw
In reply to Martin Bogomolni

On Wed, 16 Feb 2005, Martin Bogomolni wrote:

[SNIPPED...]

Quoted 4 lines

after the 'find' command is run. malloc( ) fails to allocate
afterwards. so the kernel does not free any of the missing RAM for malloc( ).

Whatever program is using malloc() is either corrupting its buffers or has a memory leak.

Malloc() will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pages malloc() "promised".

When you look at kernel memory after using find everything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes.

So, the bottom line is, if malloc() returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting.

[SNIPPED...]

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#3 lore raw

Dick,

I should say that the malloc() succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer.

After taking another look at the problem, the kernel has a lot of
memory tied up in the inode and dentry cache. I've tuned
/proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio with no real success in shrinking the amount of memory used by these caches.

Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it?

-Martin

Quoted 34 lines · linux-os

On Wed, 16 Feb 2005, Martin Bogomolni wrote:

[SNIPPED...]

Quoted 4 lines

after the 'find' command is run. malloc( ) fails to allocate
afterwards. so the kernel does not free any of the missing RAM for malloc( ).

Whatever program is using malloc() is either corrupting its buffers or has a memory leak.

Malloc() will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pages malloc() "promised".

When you look at kernel memory after using find everything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes.

So, the bottom line is, if malloc() returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting.

[SNIPPED...]

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#4 lore raw

kernel 2.4 inode/dentry cache not clearing on umount?

Also .. David :

Are you saying that, on a system with 256Megs of ram, of which the kernel is reporting only 3-4Mb free because the inode/dentry caches are taking up most of the memory, and NO page/swap file....

char *p;
p = (char *) malloc( 64*1024*1024 );

I assure you that under these conditions, the malloc( ) will fail with NULL.

---------------------------------

Now, in the meantime I have discovered that merely unmounting the filesystem is not enough to clear the dcache and icache.

However, if I unmount the filesystem then run:

cat /dev/hda > /dev/null

This causes the inode/dentry cache to finally shrink and the amount of
available free memory increases back to ~200Mb. However, this
reduction does not immediately take place when the filesystem is unmounted, and while the filesystem is mounted .. the inode/dentry cache does not shrink and leaves only 3Mb of available free memory.

Quoted 57 lines · Martin Bogomolni

Dick,

I should say that the malloc() succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer.

After taking another look at the problem, the kernel has a lot of
memory tied up in the inode and dentry cache. I've tuned
/proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio with no real success in shrinking the amount of memory used by these caches.

Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it?

-Martin

Quoted 34 lines · linux-os

On Wed, 16 Feb 2005, Martin Bogomolni wrote:

[SNIPPED...]

Quoted 4 lines

after the 'find' command is run. malloc( ) fails to allocate
afterwards. so the kernel does not free any of the missing RAM for malloc( ).

Whatever program is using malloc() is either corrupting its buffers or has a memory leak.

Malloc() will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pages malloc() "promised".

When you look at kernel memory after using find everything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes.

So, the bottom line is, if malloc() returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting.

[SNIPPED...]

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#5 lore raw

kernel 2.4 inode/dentry cache not clearing on umount?

Also .. David :

Are you saying that, on a system with 256Megs of ram, of which the kernel is reporting only 3-4Mb free because the inode/dentry caches are taking up most of the memory, and NO page/swap file....

char *p;
p = (char *) malloc( 64*1024*1024 );

I assure you that under these conditions, the malloc( ) will fail with NULL.

---------------------------------

Now, in the meantime I have discovered that merely unmounting the filesystem is not enough to clear the dcache and icache.

However, if I unmount the filesystem then run:

cat /dev/hda > /dev/null

This causes the inode/dentry cache to finally shrink and the amount of
available free memory increases back to ~200Mb. However, this
reduction does not immediately take place when the filesystem is unmounted, and while the filesystem is mounted .. the inode/dentry cache does not shrink and leaves only 3Mb of available free memory.

#6 lore raw
In reply to Martin Bogomolni
Quoted 11 lines · Martin Bogomolni

I should say that the malloc() succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer.

After taking another look at the problem, the kernel has a lot of
memory tied up in the inode and dentry cache. I've tuned
/proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio with no real success in shrinking the amount of memory used by these caches.

Quoted 3 lines

Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it?

Are you using the old (1.x) or new (2.x) ntfs driver?

If you are using the old one you could try using the new driver. That should be a lot better than the old one in terms of how much memory it uses. You can get an outdated patch (but should still work) for the new driver here:

http://sourceforge.net/project/showfiles.php?group_id=13956&package_id=21892

Best regards,

Anton

Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/