Skip to contentSkip to conversation

buffer leakage in kernel?

DISCUSSION Roman Dementiev 4 participants 6 messages Jan 24, 2003 – Feb 4, 2003 VFS

lore
Share & tools
lore
#0 lore raw

Hello everyone,

I've met with following problem (kernel 2.4.20-pre4 ):
I write and read sequentially from/to 8 files each of 64 Gbytes (not a mistake, 64 Gbyte),
each on different disk. The files are opened with flag O_DIRECT. I have 1 Gbyte RAM, no swap.
While this scanning is running, number of "buffers" reported by ''free" and in /proc/meminfo
is continuously increasing up to ~ 500 MB !! When the program exits normally or I break it, number
of "buffers" does not decrease and even increases if I do operations on other files.

This is not nice at all when I have another applications running with memory consumption > 500 MB: when my "scanner" approaches 50G border on
each disk, I've got numerous "Out of memory" murders :(. Even 'ssh' to this machine
is killed :(

Could anyone explain why it happens? I suppose that it is a memory leakage in
file system buffer management.
Is it fixed already in any patch?

Bye,
Roman

#1 lore raw
Quoted 12 lines · Roman Dementiev

I've met with following problem (kernel 2.4.20-pre4 ):
I write and read sequentially from/to 8 files each of 64 Gbytes (not a mistake, 64 Gbyte),
each on different disk. The files are opened with flag O_DIRECT. I have 1 Gbyte RAM, no swap.
While this scanning is running, number of "buffers" reported by ''free" and in /proc/meminfo
is continuously increasing up to ~ 500 MB !! When the program exits normally or I break it, number
of "buffers" does not decrease and even increases if I do operations on other files.

Quoted 4 lines

This is not nice at all when I have another applications running with memory consumption > 500 MB: when my "scanner" approaches 50G border on each disk, I've got numerous "Out of memory" murders :(. Even 'ssh' to this machine is killed :(

There was a bug in vanilla 2.4.18 related to O_DIRECT that is fixed in the RH kernel, but I don't know when/if it was merged into the main kernel. As always, testing with the most recent kernel (2.4.21-pre3 currently) will tell you whether that bug has been fixed already or not.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#2 lore raw
In reply to Roman Dementiev
Quoted 11 lines · Roman Dementiev

Hello everyone,

I've met with following problem (kernel 2.4.20-pre4 ):
I write and read sequentially from/to 8 files each of 64 Gbytes (not a mistake, 64 Gbyte),
each on different disk. The files are opened with flag O_DIRECT. I have 1 Gbyte RAM, no swap.
While this scanning is running, number of "buffers" reported by ''free" and in /proc/meminfo
is continuously increasing up to ~ 500 MB !!

You did not specify the filesystem type. I shall assume ext2.

This buffer growth is expected - ext2 uses one 4k indirect block to describe the disk location of 4M of file data. Those indirect blocks are cached, and appear as "buffers" in the memory accounting.

So after having read 500G of ext2 file with O_DIRECT, you would expect there to be 500M of indirects in the block device pagecache. (We shouldn't be calling this "buffers" any more. That is inaccurate, and confuses people into thinking that Linux has a buffer cache).

Quoted 4 lines

When the program exits
normally or I break it, number
of "buffers" does not decrease and even increases if I do operations on other files.

O_DIRECT operations, yes. If you were to read one of these files without O_DIRECT, you should see "Buffers" decreasing as they are reclaimed to make way for the newly introduced file cache.

Quoted 9 lines

This is not nice at all when I have another applications running with memory consumption > 500 MB: when my "scanner" approaches 50G border on
each disk, I've got numerous "Out of memory" murders :(. Even 'ssh' to this machine
is killed :(

Could anyone explain why it happens? I suppose that it is a memory leakage in file system buffer management.

Sounds like a bug.

Are you reading these large files via a single application, or via one process per file?

How large is the buffer into which the application is performing the O_DIRECT read?

Please perform this test:

1: Wait until you have 500M "Buffers"
2: cat 64_gig_file > /dev/null
3: Now see how large "Buffers" is. It should have reduced a lot.

#3 lore raw
Quoted 5 lines · Andrew Morton
Quoted 3 lines

While this scanning is running, number of "buffers" reported by ''free" and in /proc/meminfo
is continuously increasing up to ~ 500 MB !!

You did not specify the filesystem type. I shall assume ext2.

right

Quoted 9 lines
Quoted 9 lines

This is not nice at all when I have another applications running with memory consumption > 500 MB: when my "scanner" approaches 50G border on
each disk, I've got numerous "Out of memory" murders :(. Even 'ssh' to this machine
is killed :(

Could anyone explain why it happens? I suppose that it is a memory leakage in file system buffer management.

Thanx for explanation.

Quoted 4 lines

Sounds like a bug.

Are you reading these large files via a single application, or via one process per file?

via single application with 8 I/O threads + 1 thread.

Quoted 2 lines

How large is the buffer into which the application is performing the O_DIRECT read?

Application allocates 512 MB and never uses more.
2MB buffers are used for each read() and write() calls.
Each file has only one read or write request going any time. There is no othr memory-
hungry applications running.

Quoted 5 lines

Please perform this test:

1: Wait until you have 500M "Buffers"
2: cat 64_gig_file > /dev/null
3: Now see how large "Buffers" is. It should have reduced a lot.

Yes, it worked, they had reduced.
Does this mean, that cached indirect buffers can't be kicked out of memory automatically
and ONLY non-O_DIRECT access can do it? I suppose, they should be displaced by newly allocated indirect buffers and user memory allocation.

Roman

#4 lore raw
Quoted 6 lines · Roman Dementiev

Application allocates 512 MB and never uses more.
2MB buffers are used for each read() and write() calls.
Each file has only one read or write request going any time. There is no othr memory-
hungry applications running.

OK.

Quoted 11 lines
Quoted 5 lines

Please perform this test:

1: Wait until you have 500M "Buffers"
2: cat 64_gig_file > /dev/null
3: Now see how large "Buffers" is. It should have reduced a lot.

Yes, it worked, they had reduced.
Does this mean, that cached indirect buffers can't be kicked out of memory automatically
and ONLY non-O_DIRECT access can do it? I suppose, they should be displaced by newly allocated indirect buffers and user memory allocation.

I suspect what is happening is that you've managed to find a code path in which the kernel is allocating lots of memory in a mode in which it cannot run effective page reclaim. This would be more likely to be true if you only see the failures when writing.

It would help if you could change mm/vmscan.c:try_to_free_pages_zone() thusly:

/*
* Hmm.. Cache shrink failed - time to kill something?
* Mhwahahhaha! This is the part I really like. Giggle. */

+	show_stack(0);
        out_of_memory();
        return 0;

}

and pass the resulting log output through ksyoops.

And here's a protopatch to teach the kernel to make sure that there's a decent amount of free memory before it goes and performs GFP_NOFS pagecache allocations:

fs/buffer.c renamed from fs/buffer.c~a +2 -0
diff -puN fs/buffer.c~a fs/buffer.c
@@ -2112,6 +2112,8 @@ int generic_direct_IO(int rw, struct ino
 	for (i = 0; i < nr_blocks; i++, blocknr++) {
 		struct buffer_head bh;
 
+		try_to_free_pages(GFP_KERNEL);
+
 		bh.b_state = 0;
 		bh.b_dev = inode->i_dev;
 		bh.b_size = blocksize;

If that sheds no light, please send me the app and I'll see if I can reproduce it. Thanks.

#5 lore raw
Quoted 38 lines · Andrew Morton

I suspect what is happening is that you've managed to find a code path in which the kernel is allocating lots of memory in a mode in which it cannot run effective page reclaim. This would be more likely to be true if you only see the failures when writing.

It would help if you could change mm/vmscan.c:try_to_free_pages_zone() thusly:

/*
* Hmm.. Cache shrink failed - time to kill something?
* Mhwahahhaha! This is the part I really like. Giggle. */

(quoted diff) +1 -0
+	show_stack(0);
        out_of_memory();
        return 0;
 }

and pass the resulting log output through ksyoops.

And here's a protopatch to teach the kernel to make sure that there's a decent amount of free memory before it goes and performs GFP_NOFS pagecache allocations:

diff -puN fs/buffer.c~a fs/buffer.c

fs/buffer.c renamed from fs/buffer.c~a +2 -0
@@ -2112,6 +2112,8 @@ int generic_direct_IO(int rw, struct ino
 	for (i = 0; i < nr_blocks; i++, blocknr++) {
 		struct buffer_head bh;
 
+		try_to_free_pages(GFP_KERNEL);
+
 		bh.b_state = 0;
 		bh.b_dev = inode->i_dev;
 		bh.b_size = blocksize;
Quoted 4 lines

If that sheds no light, please send me the app and I'll see if I can reproduce it. Thanks.

I have not yet tried that because of the lack of time. Instead I installed 2.4.21-pre4. The problem seems to be fixed there.

Roman