writepage return value check in vmscan.c
Hi Andrew,
It might be a silly question.
Is it a bug in vmscan.c that it do not check the return value of writepage when shrinking the cache?
Some background of the problem:
I was tracing some heavy swapping problem relate to high memory usage when running many virtual machines in VMware GSX server some time ago. If the total ram size of different virtual machine add up to some level, bigger than 2G on a 4Gmachine.Kernel will keep swapping and no response.
VMware open a tmp ram file for each virtual machine and mmap on that file to share memory between different process.
I soon find out the hidden size limit on shmfs even though the ram file is open on a ext2/ext3 file system. If I change the ram file open on /dev/shm then the problem seems to go away. I guess that is because /tmp directory is full but bigger /tmp did not help. that is another issue though.
I try to find out what happen if user memory map a sparse file then kernel try to write it back to disk and hit a no disk space error. To my surprise, it seems to me that both 2.4 and 2.5 kernel do not check the return value of "writepage". If there is an error like ENOSPC it will just drop it on the ground? Do I miss something obvious?
BTW, I am amazed that there is so many way user can abuse the mmap system call. e.g. open a file, ftruncate to a bigger size, unlink that file while keep the file descriptor, mmap to some memory using that descriptor, close that descriptor, you can still use that mmaped memory.
Cheers,
Chris
Quoted 4 lines · [email protected] wrote:
Hi Andrew,
It might be a silly question.
It's an excellent question.
Quoted 6 lines
...
I try to find out what happen if user memory map a sparse file then kernel try to write it back to disk and hit a no disk space error. To my surprise, it seems to me that both 2.4 and 2.5 kernel do not check the return value of "writepage". If there is an error like ENOSPC it will just drop it on the ground? Do I miss something obvious?
Yup. If the kernel cannot write back your MAP_SHARED page due to ENOSPC it throws your data away.
The alternative would be to allow you to pin an arbitrary amount of unpageable memory.
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
Quoted 4 lines
BTW, I am amazed that there is so many way user can abuse the mmap system call. e.g. open a file, ftruncate to a bigger size, unlink that file while keep the file descriptor, mmap to some memory using that descriptor, close that descriptor, you can still use that mmaped memory.
Ayup. MAP_SHARED is a crock. If you want to write to a file, use write().
View MAP_SHARED as a tool by which separate processes can attach to some shared memory which is identified by the filesystem namespace. It's not a very good way of performing I/O.
That's not to say that you deserve to have the kernel silently throw your data away as punishment for having used it though. Thanks for the prod. We should do something about that.
Quoted 3 lines · Andrew Morton
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
You still have to handle the situation where the page goes walkies and you get ENOSPC or any other ERANDOMSUPRISE from things like NFS. SIGBUS appears the right thing to do.
Quoted 8 lines · Alan Cox
Quoted 3 lines · Andrew Morton
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
You still have to handle the situation where the page goes walkies and you get ENOSPC or any other ERANDOMSUPRISE from things like NFS. SIGBUS appears the right thing to do.
I would tend to agree SIGBUS could be the right thing to do since the other (current) option is silent data corruption.
Andrea
Quoted 7 lines · Andrea Arcangeli
Quoted 5 lines · Alan Cox
Quoted 3 lines · Andrew Morton
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
Quoted 7 lines
Quoted 3 lines
You still have to handle the situation where the page goes walkies and you get ENOSPC or any other ERANDOMSUPRISE from things like NFS. SIGBUS appears the right thing to do.
I would tend to agree SIGBUS could be the right thing to do since the other (current) option is silent data corruption.
Or at least remember the data loss within the mapping for a subsequent msync/fsync operation.
We'd need a similar thing for detecting write I/O errors too.
write(fd, data); sleep(60); fsync(fd); -> doesn't report write errors.
But that's all filed under "bug fixes" and can be done after you-know-when.
Quoted 9 lines · Alan Cox
Quoted 3 lines · Andrew Morton
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
You still have to handle the situation where the page goes walkies and you get ENOSPC or any other ERANDOMSUPRISE from things like NFS. SIGBUS appears the right thing to do.
exactly. It need to preserve the page dirty in the first place.
Chris
Quoted 1 line · Andrew Morton
prod. We should do something about that.
you need to preallocate the file, then to mmap it. If you do, the kernel won't throw the data away. So the fix for vmware is to preallocate the file and later to mmap it. This way you will be notified by -ENOSPC if you run out of disk/shmfs space. Other than this I'm not so against the MAP_SHARED like Andrew, the reason the API is not so clean is that we cannot have an API at all inside a page fault to notify userspace that the ram modifications cannot be written to disk. the page fault must be transparent, there's no retvalue, so if you run out of disk space during the page fault, the page fault cannot easily tell userspace. As said the fix is very easy and consists in preallocating the space on disk (I understand that on shmfs it may not be extremely desiderable since you may prefer to defer allocation lazily to when you will need the memory but assuming your allocations are worthwhile it won't make difference after a few minutes/hours of usage and this way you will trap the -ENOSPC).
As for the task being able to reference a deleted file in memory, that's true for many other scenarios (the user could leak space by keeping the fd open and unlinking the file and at the same time to alloc lots of ram with malloc, the result would be similar), and that's why root will have to kill these malicious tasks in order to reclaim ram and disk space.
Andrea
Quoted 16 lines · Andrea Arcangeli
On Thu, Oct 24, 2002 at 01:36:43AM -0700, Andrew Morton wrote:
you need to preallocate the file, then to mmap it. If you do, the kernel won't throw the data away. So the fix for vmware is to preallocate the file and later to mmap it. This way you will be notified by
-ENOSPCif you run out of disk/shmfs space. Other than this I'm not so against theMAP_SHAREDlike Andrew, the reason the API is not so clean is that we cannot have an API at all inside a page fault to notify userspace that the ram modifications cannot be written to disk. the page fault must be transparent, there's no retvalue, so if you run out of disk space during the page fault, the page fault cannot easily tell userspace. As said the fix is very easy and consists in preallocating the space on disk (I understand that on shmfs it may not be extremely desiderable since you may prefer to defer allocation lazily to when you will need the memory but assuming your allocations are worthwhile it won't make difference after a few minutes/hours of usage and this way you will trap the-ENOSPC).
But preallocate the vmware ram file on disk is too expensive. It will slow down the guest OS boot up a lot. Many user measure how fast vmware is by counting how many seconds it takes to boot a windows guest for example. For those virtual machine which have 2G or ram, how long does it take to write a file with 2G of data?
Quoted 6 lines
As for the task being able to reference a deleted file in memory, that's true for many other scenarios (the user could leak space by keeping the fd open and unlinking the file and at the same time to alloc lots of ram with malloc, the result would be similar), and that's why root will have to kill these malicious tasks in order to reclaim ram and disk space.
vmware is definitely one of those malicious task ;-)
Chris
Quoted 18 lines · On Thu, Oct 24, 2002 at 11:30:24AM -0700, [email protected] wrote:
Quoted 16 lines · Andrea Arcangeli
On Thu, Oct 24, 2002 at 01:36:43AM -0700, Andrew Morton wrote:
you need to preallocate the file, then to mmap it. If you do, the kernel won't throw the data away. So the fix for vmware is to preallocate the file and later to mmap it. This way you will be notified by
-ENOSPCif you run out of disk/shmfs space. Other than this I'm not so against theMAP_SHAREDlike Andrew, the reason the API is not so clean is that we cannot have an API at all inside a page fault to notify userspace that the ram modifications cannot be written to disk. the page fault must be transparent, there's no retvalue, so if you run out of disk space during the page fault, the page fault cannot easily tell userspace. As said the fix is very easy and consists in preallocating the space on disk (I understand that on shmfs it may not be extremely desiderable since you may prefer to defer allocation lazily to when you will need the memory but assuming your allocations are worthwhile it won't make difference after a few minutes/hours of usage and this way you will trap the-ENOSPC).
Quoted 5 lines
But preallocate the vmware ram file on disk is too expensive. It will slow down the guest OS boot up a lot. Many user measure how fast vmware is by counting how many seconds it takes to boot a windows guest for example. For those virtual machine which have 2G or ram, how long does it take to write a file with 2G of data?
unfortunately I see no way around it and patching the kernel to loop forever on dirty pages that may never be possible to write doesn't look safe. You could check the free space on the fs and bug the user if it has less than 2G free (still it's not 100% reliable, it's a racy check, but you could also add a 100% reliable option that slowdown the startup of the vm but that guarantees no corruption can happen).
Furthmore if your machine has 2G of data you're likely to have >2G of ram and shmfs should be quick allocating 2G in such case, maybe 2/3 seconds?
Andrea
Quoted 6 lines · Andrea Arcangeli
unfortunately I see no way around it and patching the kernel to loop forever on dirty pages that may never be possible to write doesn't look safe. You could check the free space on the fs and bug the user if it has less than 2G free (still it's not 100% reliable, it's a racy check, but you could also add a 100% reliable option that slowdown the startup of the vm but that guarantees no corruption can happen).
We need space allocation. Not just for this (probably rare) case, but also for the more generic optimisation of delayed allocation.
cheers,
Rik
Signature · 4 lines
A: No. Q: Should I include quotations after my reply? http://www.surriel.com/ http://distro.conectiva.com/
Quoted 10 lines · Rik van Riel
Quoted 6 lines · Andrea Arcangeli
unfortunately I see no way around it and patching the kernel to loop forever on dirty pages that may never be possible to write doesn't look safe. You could check the free space on the fs and bug the user if it has less than 2G free (still it's not 100% reliable, it's a racy check, but you could also add a 100% reliable option that slowdown the startup of the vm but that guarantees no corruption can happen).
Quoted 3 lines
We need space allocation. Not just for this (probably rare) case, but also for the more generic optimisation of delayed allocation.
Well that's certainly the other option. Dig out the old a_ops->reservepage stuff.
It _was_ Halloween 2003, wasn't it??
It would only work for filesystems which implement reservation though, and iirc there were nasty problems doing delayed allocation in, for example, ext3. I guess ext3 would have to
reserve journal space as well as disk space. ext2 delayed
allocation was pretty straightforward though, and most of the infrastructure which it needed is there now. The replacement of buffer-based writeback with page-based, mainly.
Quoted 6 lines · Andrew Morton
Yup. If the kernel cannot write back your
MAP_SHAREDpage due to ENOSPC it throws your data away.The alternative would be to allow you to pin an arbitrary amount of unpageable memory.
I know the error handling in mmaped memory is poor. But I am not talking about that one. There are two place the mmaped memory can flush back to disk. The one you are talking about is filemap_fdatasync() in filemap.c. It will be called when try to unmmap or sync back to the disk. It at least check the err when writepage fail. But it still clear the page dirty anyway, looks bad to me.
The one I am complaining about is in vmscan.c, when kswapd try to shink the cache. Correct me if I am wrong. kswapd will flush some mmaped page back to disk to release some page from page cache. Even when write page fail. Kernel will still do:
ClearPageDirty(page);
SetPageLaunder(page);
for that page. So this page will be able to used by other process. When it have a missing page fault, it will read back the WRONG data from the disk and cause a memory corruption.
So I am expecting something like tihs:
if ((gfp_mask & __GFP_FS) && writepage) {+ unsigned long flags = page->flags;
ClearPageDirty(page); SetPageLaunder(page); page_cache_get(page); spin_unlock(&pagemap_lru_lock);
- writepage(page);
+ if (writepage(page))
+ page->flags = flags;
page_cache_release(page);
spin_lock(&pagemap_lru_lock);
continue;
}Quoted 4 lines
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
I am not sure how the user program handle that signal...
Quoted 6 lines
Ayup.
MAP_SHAREDis a crock. If you want to write to a file, usewrite().View
MAP_SHAREDas a tool by which separate processes can attach to some shared memory which is identified by the filesystem namespace. It's not a very good way of performing I/O.
That is exactly the case for vmware ram file. VMware only use it to share memory. Those are the virtual machine's memory. We don't want to write it back to disk and we don't care what is left on the file system because when vmware exit, we will throw the guest ram data away just like a real machine power off ram will lost. We are not talking about machine using flash ram :-).
It is kswapd try to flush the data and it should take response to handle the error. If it fail, one thing it should do is keep the page dirty if write back fail. At least not corrupt memory like that.
If we can deliver the error to user program that would be a plus. But this need to be fix frist.
Chris
Quoted 12 lines · On Thu, Oct 24, 2002 at 10:57:18AM -0700, [email protected] wrote:
if ((gfp_mask & __GFP_FS) && writepage) {+ unsigned long flags =
page->flags;ClearPageDirty(page); SetPageLaunder(page); page_cache_get(page); spin_unlock(&pagemap_lru_lock);(quoted diff) +2
-1- writepage(page); + if (writepage(page)) + page->flags = flags;
Quoted 5 lines
page_cache_release(page);
spin_lock(&pagemap_lru_lock);
continue;
}side note, you should use atomic bitflag operations here or you risk to lose a bit set by another cpu between the read and the write. you basically meant SetPageDirty() if writepage fails. That is supposed to happen in the lowlevel layer (like in fail_writepage) but the problem here is that this isn't ramfs, and block_write_full_page could left locked in ram lots of pages if it would disallow these pages to be discared from the vm.
Quoted 20 lines
Quoted 3 lines
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
I am not sure how the user program handle that signal...
Quoted 6 lines
Ayup.
MAP_SHAREDis a crock. If you want to write to a file, usewrite().View
MAP_SHAREDas a tool by which separate processes can attach to some shared memory which is identified by the filesystem namespace. It's not a very good way of performing I/O.That is exactly the case for vmware ram file. VMware only use it to share memory. Those are the virtual machine's memory. We don't want to write it back to disk and we don't care what is left on the file system because when vmware exit, we will throw the guest ram data away just like a real machine power off ram will lost. We are not talking about machine using flash ram :-).
Quoted 6 lines
It is kswapd try to flush the data and it should take response to handle the error. If it fail, one thing it should do is keep the page dirty if write back fail. At least not corrupt memory like that.
If we can deliver the error to user program that would be a plus. But this need to be fix frist.
as said this cannot be fixed easily in kernel, or it would be trivial to lockup a machine by filling the fs changing the i_size of a file and by marking all ram in the machine dirty in the hole, the vm must be allowed to discard those pages and invaliding those posted writes. At least until a true solution will be available you should change vmware to preallocate the file, then it will work fine because you will catch the ENOSPC error during the preallocation. If you work on shmfs that will be very quick indeed.
Andrea
Quoted 13 lines · Andrea Arcangeli
Quoted 12 lines · On Thu, Oct 24, 2002 at 10:57:18AM -0700, [email protected] wrote:
if ((gfp_mask & __GFP_FS) && writepage) {+ unsigned long flags =
page->flags;ClearPageDirty(page); SetPageLaunder(page); page_cache_get(page); spin_unlock(&pagemap_lru_lock);(quoted diff) +2
-1- writepage(page); + if (writepage(page)) + page->flags = flags;
Quoted 8 lines
Quoted 5 lines
page_cache_release(page); spin_lock(&pagemap_lru_lock); continue; }side note, you should use atomic bitflag operations here or you risk to lose a bit set by another cpu between the read and the write. you
Thanks. I am just shooting in dark.
Quoted 5 lines
basically meant
SetPageDirty()if writepage fails. That is supposed to happen in the lowlevel layer (like infail_writepage) but the problem here is that this isn't ramfs, andblock_write_full_pagecould left locked in ram lots of pages if it would disallow these pages to be discared from the vm.
Exactly.
Quoted 25 lines
Quoted 25 lines
Quoted 4 lines
A few fixes have been discussed. One way would be to allocate the space for the page when it is first faulted into reality and deliver SIGBUS if backing store for it could not be allocated.
I am not sure how the user program handle that signal...
Quoted 6 lines
Ayup.
MAP_SHAREDis a crock. If you want to write to a file, usewrite().View
MAP_SHAREDas a tool by which separate processes can attach to some shared memory which is identified by the filesystem namespace. It's not a very good way of performing I/O.That is exactly the case for vmware ram file. VMware only use it to share memory. Those are the virtual machine's memory. We don't want to write it back to disk and we don't care what is left on the file system because when vmware exit, we will throw the guest ram data away just like a real machine power off ram will lost. We are not talking about machine using flash ram :-).
It is kswapd try to flush the data and it should take response to handle the error. If it fail, one thing it should do is keep the page dirty if write back fail. At least not corrupt memory like that.
Quoted 6 lines
Quoted 2 lines
If we can deliver the error to user program that would be a plus. But this need to be fix frist.
as said this cannot be fixed easily in kernel, or it would be trivial to lockup a machine by filling the fs changing the
i_sizeof a file and by marking all ram in the machine dirty in the hole, the vm must be allowed
Yes, but even now days it will able to lockup machine by doing that.
Try the test bigmm program I attach to this mail. It will simulate vmware's memory mapping. It can easily lockup the machine even though there is enough disk space.
See the comment at the source for parameter. basically, if you want 3 virtual machine, each have 2 process, using 1 G ram each you can do:
bigmm -i 3 -t 2 -c 1024
I run it on two 4G and 8G smp machine. Both can dead lock if I mmap enough memory.
I haven't try it on the latest kernel yet. But last time I try it, it works every time. I have to reset the machine. I mean ram file create on normal file system.
But if I create it on /dev/shm, the kernel can correctly kill some of the process and free the memory.
Prepare to reset the machine if you try that, you have been warned :-)
Quoted 5 lines
to discard those pages and invaliding those posted writes. At least until a true solution will be available you should change vmware to preallocate the file, then it will work fine because you will catch the ENOSPC error during the preallocation. If you work on shmfs that will be very quick indeed.
Yes, shmfs seems to be the only choice so far.
Chris
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
int pagesize = 4096;
int instance = 1;
int thread = 1;
int blocksize = 1024*1024; // 1M
int blocks = 1024; // total 1 G
char *filename="myram";
char *memblock[1024*256];
int memmap(char *argv[], int ppid)
{
int i,j,fd,err=0,pid,k;
char file[256];
sprintf(argv[0],"a%d",ppid);
sprintf(file,"%s.%d",filename,ppid);
fd = open(file,O_CREAT|O_TRUNC|O_RDWR,00644);
if (fd<0) {
perror("open");
err = fd;
goto exit;
}
err = unlink(file);
if (err<0) {
perror("unlink");
goto exit_close;
}
err = ftruncate(fd,blocksize*blocks);
if (err<0) {
perror("ftruncate");
goto exit_close;
}
/* fork more process(thread) to use same share mem file */
for (i=1;i<thread;i++) {
pid = fork();
if (pid <0) {
perror("fork");
exit(1);
}
if(pid) {
printf("%d fork child %d\n",ppid,pid);
}else {
sprintf(argv[0],"a%d.%d",ppid,i);
break;
}
}
for (i=0;i<blocks;i++) {
void * ptr;
ptr = mmap(NULL,blocksize, PROT_READ|PROT_WRITE,
MAP_SHARED, fd, i*blocksize);
if (ptr==MAP_FAILED) {
perror("mmap");
break;
}
memblock[i] = ptr;
}
/* touch all the share memory like a guest vm */
while (1) {
sleep(1);
for (i=0;i<blocks;i++)
for (j=0;j<blocksize;j+=pagesize)
memblock[i][j]=i&0xff;
}
err = 0;
exit_close:
close(fd);
exit:
return err;
}int main (int argc, char *argv[])
{
int i,pid;
int c;
while (1) {
c = getopt(argc,argv,"b:c:i:t:");
if (c<=0) break;
switch (c) {
case 'i':
/* number of virtual machine */
instance = strtol(optarg,NULL,0);
break;
case 't':
/* number of process per virtual machine */
thread = strtol(optarg,NULL,0);
break;
case 'c':
/* number of memory blocks in each virtual machine */
blocks = strtol(optarg,NULL,0);
break;
case 'b':
/* size of a memory block */
blocksize = strtol(optarg,NULL,0);
break;
}
}
printf("i:%d t:%d b:%d c:%d\n",instance,thread,blocksize,blocks);
for (i=0;i<instance;i++) {
pid = fork();
if (pid <0) {
perror("mainfork");
exit(1);
}
if(pid) {
printf("mainfork child %d\n",pid);
}else {
return memmap(argv,i);
}
}
return 0;
}Quoted 10 lines · [email protected] wrote:
...
See the comment at the source for parameter. basically, if you want 3 virtual machine, each have 2 process, using 1 G ram each you can do:bigmm -i 3 -t 2 -c 1024
I run it on two 4G and 8G smp machine. Both can dead lock if I mmap enough memory.
Are you sure it's a deadlock? A large MAP_SHARED load like this on a 2.4 highmem machine can go into a spin, but it will come back to life after several minutes.
Quoted 6 lines · Andrew Morton
Quoted 5 lines · [email protected] wrote:
...
See the comment at the source for parameter. basically, if you want 3 virtual machine, each have 2 process, using 1 G ram each you can do:
Quoted 7 lines
Quoted 5 lines
bigmm -i 3 -t 2 -c 1024
I run it on two 4G and 8G smp machine. Both can dead lock if I mmap enough memory.
Are you sure it's a deadlock? A large
MAP_SHAREDload like this
deadlock is the wrong word. Its harddisk keep spinning and not response to anything.
Quoted 2 lines
on a 2.4 highmem machine can go into a spin, but it will come back to life after several minutes.
No, it will not come back to life, at least not after several minutes. And there is not sign it is going to come back to life.
Chris
Quoted 6 lines · [email protected] wrote:
Quoted 3 lines
on a 2.4 highmem machine can go into a spin, but it will come back to life after several minutes.
No, it will not come back to life, at least not after several minutes. And there is not sign it is going to come back to life.
A 2.5G machine would, iirc, spin for 3-5 minutes.
Umm, probably the time would increase somewhat exponentially with memory size so yes, you could be in for a very long wait.
-ac kernels have an lru per zone and so would not be bitten by this failure. If indeed you are striking this problem, which is described at
http://mail.nl.linux.org/linux-mm/2002-08/msg00049.html
Quoted 4 lines · Andrew Morton
-ac kernels have an lru per zone and so would not be bitten by this failure. If indeed you are striking this problem, which is described at
http://mail.nl.linux.org/linux-mm/2002-08/msg00049.html
Is it the 2.4 or 2.5 (or both) ac kernels that have the per zone lru? I have some stuff I'd like to try with that.
Thanks,
Paul Larson
Quoted 7 lines · Paul Larson
Quoted 4 lines · Andrew Morton
-ac kernels have an lru per zone and so would not be bitten by this failure. If indeed you are striking this problem, which is described at
http://mail.nl.linux.org/linux-mm/2002-08/msg00049.htmlIs it the 2.4 or 2.5 (or both) ac kernels that have the per zone lru? I have some stuff I'd like to try with that.
2.4-rmap, 2.4-ac and any 2.5 (including -ac).
Quoted 4 lines · Paul Larson
Quoted 1 line · Andrew Morton
Is it the 2.4 or 2.5 (or both) ac kernels that have the per zone lru? I have some stuff I'd like to try with that.
2.4-rmap
2.4-ac
2.5 all
cheers,
Rik
Signature · 3 lines
Bravely reimplemented by the knights who say "NIH". http://www.surriel.com/ http://distro.conectiva.com/ Current spamtrap: <a href=mailto:"[email protected]">[email protected]</a>
Quoted 6 lines · On Thu, Oct 24, 2002 at 12:15:32PM -0700, [email protected] wrote:
Yes, but even now days it will able to lockup machine by doing that.
Try the test bigmm program I attach to this mail. It will simulate vmware's memory mapping. It can easily lockup the machine even though there is enough disk space.
Quoted 7 lines
See the comment at the source for parameter. basically, if you want 3 virtual machine, each have 2 process, using 1 G ram each you can do:
bigmm -i 3 -t 2 -c 1024
I run it on two 4G and 8G smp machine. Both can dead lock if I mmap enough memory.
I run the above command on my laptop with 256M of ram and 1G of swap with kde running (though idle) and the task was correctly killed:
Oct 24 22:29:32 x30 kernel: VM: killing process bigmm
the machine never deadlocked. Probably it's one of the oom deadlocks that I fixed in my 2.4 -aa tree and that the oom killer heuristic in mainline cannot figure out. Please try to reproduce with 2.4.20pre11aa1. thanks.
Quoted 1 line
Prepare to reset the machine if you try that, you have been warned :-)
If you're running an oom deadlock prone kernel.
Quoted 7 lines
Quoted 5 lines
to discard those pages and invaliding those posted writes. At least until a true solution will be available you should change vmware to preallocate the file, then it will work fine because you will catch the ENOSPC error during the preallocation. If you work on shmfs that will be very quick indeed.
Yes, shmfs seems to be the only choice so far.
Agreed.
Andrea
Quoted 10 lines · Andrea Arcangeli
Quoted 9 lines · On Thu, Oct 24, 2002 at 12:15:32PM -0700, [email protected] wrote:
Yes, but even now days it will able to lockup machine by doing that.
Try the test bigmm program I attach to this mail. It will simulate vmware's memory mapping. It can easily lockup the machine even though there is enough disk space.
See the comment at the source for parameter. basically, if you want 3 virtual machine, each have 2 process, using 1 G ram each you can do:
Quoted 7 lines
Quoted 4 lines
bigmm -i 3 -t 2 -c 1024
I run it on two 4G and 8G smp machine. Both can dead lock if I mmap enough memory.
I run the above command on my laptop with 256M of ram and 1G of swap with kde running (though idle) and the task was correctly killed:
That is probably too small ram to start with. What can you expect to ask for 3G on a 1/4 G machine?
Quoted 7 lines
Oct 24 22:29:32 x30 kernel: VM: killing process bigmmthe machine never deadlocked. Probably it's one of the oom deadlocks that I fixed in my 2.4 -aa tree and that the oom killer heuristic in mainline cannot figure out. Please try to reproduce with 2.4.20pre11aa1. thanks.
I will definitely try it when I can use that big memory machine again. Other people is running (windows) test on it right now.
Quoted 4 lines
Quoted 2 lines
Prepare to reset the machine if you try that, you have been warned :-)
If you're running an oom deadlock prone kernel.
When it dies, it is not deadlock though. Hard disk keep spinning. It looks like dead loop in swapping and not response to any thing else. But one thing for sure, OOM did not kill it correctly.
Chris
Quoted 2 lines · [email protected] wrote:
bigmm -i 3 -t 2 -c 1024
That's a nice little box killer you have there.
With mem=4G, running bigmm -i 5 -t 2 -c 1024:
2.4.19: Ran for a few minutes, got slower and slower and eventually stopped. kupdate had taken 30 seconds CPU and all CPUs were spinning in shrink_cache(). Had to reset.
2.4.20-pre8-ac1: Ran for a minute, froze up for a couple of minutes then recovered and remained comfortable.
2.5.44-mm5: had a few 0.5-second stalls in vmstat output, no other problems.
It's probably the list search failure, but I can't say for sure at this time.
Quoted 5 lines · Andrew Morton
Quoted 2 lines · [email protected] wrote:
bigmm -i 3 -t 2 -c 1024
That's a nice little box killer you have there.
Thanks. It kills on all our customer's kernel, they don't use the bleeding edge kernel at all. It is interesting to see vmware serve as some heavy load stress test tool. It will give some real world load to the OS, e.g. the load need to boot a windows etc. You can stack many of them to abuse the OS.
Quoted 9 lines
With mem=4G, running bigmm -i 5 -t 2 -c 1024:
2.4.19: Ran for a few minutes, got slower and slower and eventually stopped. kupdate had taken 30 seconds CPU and all CPUs were spinning in
shrink_cache(). Had to reset.2.4.20-pre8-ac1: Ran for a minute, froze up for a couple of minutes then recovered and remained comfortable.
How many instance of bigmm left there? It should be 10 bigmm processes before oom kickin.
Quoted 6 lines
2.5.44-mm5: had a few 0.5-second stalls in vmstat output, no other problems.
It's probably the list search failure, but I can't say for sure at this time.
Chris
Quoted 6 lines · [email protected] wrote:
Quoted 4 lines · Andrew Morton
Quoted 2 lines · [email protected] wrote:
bigmm -i 3 -t 2 -c 1024
Quoted 7 lines
Quoted 1 line
That's a nice little box killer you have there.
Thanks. It kills on all our customer's kernel, they don't use the bleeding edge kernel at all. It is interesting to see vmware serve as some heavy load stress test tool. It will give some real world load to the OS, e.g. the load need to boot a windows etc. You can stack many of them to abuse the OS.
I tested Andrea's latest kernel. It survived.
Probably because it left 100 megabytes of lowmem unallocated throughout the test.
Quoted 7 lines
Quoted 7 lines
With mem=4G, running bigmm -i 5 -t 2 -c 1024:
2.4.19: Ran for a few minutes, got slower and slower and eventually stopped. kupdate had taken 30 seconds CPU and all CPUs were spinning in
shrink_cache(). Had to reset.
Quoted 5 lines
Quoted 2 lines
2.4.20-pre8-ac1: Ran for a minute, froze up for a couple of minutes then recovered and remained comfortable.
How many instance of bigmm left there? It should be 10 bigmm processes before oom kickin.
Well, they should all be left running? All this memory has file-backing, and is easily reclaimable.
umm, yes. There could be bogus oom-killings in the combined-LRU VMs. But I saw none in testing.
All of which is great fun, but it leaves open the question "what the heck can vmware do about it". I wish there was a clear answer.
If the customer is running a suse/UL kernel they're presumably OK.
If their kernel comes from kernel.org they should add Andrea's patch. Which means they get an absolute boatload of stuff which they may not want:
1223 files changed, 306053 insertions(+), 9655 deletions(-)
but that kernel performs well.
If they're running an RH-rmap kernel then they're probably okayish, although I'd recommend more testing there.
If they're running an RHAS-style kernel then I do not know. It may fail.
Quoted 11 lines · Andrew Morton
Quoted 8 lines · [email protected] wrote:
Quoted 5 lines · Andrew Morton
Quoted 2 lines · [email protected] wrote:
bigmm -i 3 -t 2 -c 1024
That's a nice little box killer you have there.
I tested Andrea's latest kernel. It survived.
great. I can try the hundred-win2k-vm test on linux some day and see what happen.
Quoted 11 lines
Probably because it left 100 megabytes of lowmem unallocated throughout the test.
Quoted 7 lines
Quoted 7 lines
With mem=4G, running bigmm -i 5 -t 2 -c 1024:
2.4.19: Ran for a few minutes, got slower and slower and eventually stopped. kupdate had taken 30 seconds CPU and all CPUs were spinning in
shrink_cache(). Had to reset.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
2.4.20-pre8-ac1: Ran for a minute, froze up for a couple of minutes then recovered and remained comfortable.
How many instance of bigmm left there? It should be 10 bigmm processes before oom kickin.
Well, they should all be left running? All this memory has file-backing, and is easily reclaimable.
In my experiment, if some bigmm get killed by oom killer.
Quoted 8 lines
umm, yes. There could be bogus oom-killings in the combined-LRU VMs. But I saw none in testing.
All of which is great fun, but it leaves open the question "what the heck can vmware do about it". I wish there was a clear answer.
We told them to dump the ram file at /dev/shm and prepare a large swap space. And be gentle to linux, don't push vm's total ram beyond what is on the machine. It is running OK so far.
We did try to recomment some SuSE or Redhat new kernel. It did not work at that time. But I am not surprised at all, it is something like 2.4.7 based. We have this problem on linux for some time now.
Quoted 8 lines
If the customer is running a suse/UL kernel they're presumably OK.
If their kernel comes from kernel.org they should add Andrea's patch. Which means they get an absolute boatload of stuff which they may not want:
1223 files changed, 306053 insertions(+), 9655 deletions(-)but that kernel performs well.
Patch kernel and ask them to build kernel is not an option at all. They just want something reliable. They trust most major linux distribution's kenrel just don't home make one.
Quoted 3 lines
If they're running an RH-rmap kernel then they're probably okayish, although I'd recommend more testing there.
That I don't know.
Quoted 3 lines
If they're running an RHAS-style kernel then I do not know. It may fail.
Chris
Quoted 15 lines · Andrew Morton
Quoted 6 lines · [email protected] wrote:
Quoted 4 lines · Andrew Morton
Quoted 2 lines · [email protected] wrote:
bigmm -i 3 -t 2 -c 1024
Quoted 7 lines
Quoted 1 line
That's a nice little box killer you have there.
Thanks. It kills on all our customer's kernel, they don't use the bleeding edge kernel at all. It is interesting to see vmware serve as some heavy load stress test tool. It will give some real world load to the OS, e.g. the load need to boot a windows etc. You can stack many of them to abuse the OS.
Quoted 4 lines
I tested Andrea's latest kernel. It survived.
Probably because it left 100 megabytes of lowmem unallocated throughout the test.
that's unrelated to the vm code though (in terms of per-zone lru mentioned by Andrew for 2.5 that in turns breaks all the aging information compared to 2.4), that is meant to definitely fix another highmem unbalance bug where all the lowmem could be pinned and made unfreeable by lowmem users despite plenty of highmem is still available. That's the fix to the google mlock deadlock, mainline has a weak attempt to fix it in another manner, but I backed it out since it's too weak to be effective in the real workloads (it didn't fix the problem in real life) and I instead kept my first approch that is THE definitive fix. btw, 2.5 has still the weak approch, so it's still subject to the google bug, I will fix it in my tree while moving to 2.5.
Quoted 1 line
If the customer is running a suse/UL kernel they're presumably OK.
Right.
Andrea
(PS about your previous comment on the swap needed: kernels <= 2.4.9
(9 != 19) also have the problem that vm+swap isn't all the vm that vmware can use, for them you need the double of swap)
Quoted 1 line · On Thu, 24 Oct 2002, [email protected] wrote:
Yes, shmfs seems to be the only choice so far.
So why don't you use Posix or SYSV shared mem?
Greetings
Christoph
They are the same as shmfs to linux kernel. Why does vmware not use it in the first place? It is possible due to some the history reason.
BTW, I have another question. For the 8G memory machine, do we need to setup 16G swap space? Think about the time it take to write 16G data, does it still make sense that swap space is twice as big as memory?
And the swap partition has limit as 2G. So we need to setup 8 swap partitions if we want 16G swap.
Thanks
Chris
Quoted 9 lines · Christoph Rohland
Quoted 1 line · On Thu, 24 Oct 2002, [email protected] wrote:
Yes, shmfs seems to be the only choice so far.
So why don't you use Posix or SYSV shared mem?
Greetings
Christoph
Quoted 7 lines · On Mon, Oct 28, 2002 at 10:44:20AM -0800, [email protected] wrote:
They are the same as shmfs to linux kernel. Why does vmware not use it in the first place? It is possible due to some the history reason.
BTW, I have another question. For the 8G memory machine, do we need to setup 16G swap space? Think about the time it take to write 16G data, does it still make sense that swap space is twice as big as memory?
swap space doesn't need to be twice as big as ram. That's fixed long ago.
swap+ram is the total amount of virtual memory that you can use in vmware.
Quoted 3 lines
And the swap partition has limit as 2G. So we need to setup 8 swap partitions if we want 16G swap.
that's a silly restriction of mkswap, the kernel doesn't care, it can handle way more than 2G (however there's an high bound at some unpractical level, to go safe the math limit should be re-encoded in mkswap, of course it changes for every arch because the pte layout is different).
Andrea
Quoted 6 lines · Andrea Arcangeli
swap space doesn't need to be twice as big as ram. That's fixed long ago.
swap+ram is the total amount of virtual memory that you can use in vmware.
Cool.
Quoted 10 lines
Quoted 4 lines
And the swap partition has limit as 2G. So we need to setup 8 swap partitions if we want 16G swap.
that's a silly restriction of mkswap, the kernel doesn't care, it can handle way more than 2G (however there's an high bound at some unpractical level, to go safe the math limit should be re-encoded in mkswap, of course it changes for every arch because the pte layout is different).
Thanks
Chris
On Mon, 28 Oct 2002, Andrea Arcangeli wrote:
| On Mon, Oct 28, 2002 at 10:44:20AM -0800, [email protected] wrote: | > They are the same as shmfs to linux kernel. Why does vmware not use it | > in the first place? It is possible due to some the history reason. | > | > BTW, I have another question. For the 8G memory machine, do we need | > to setup 16G swap space? Think about the time it take to write 16G | > data, does it still make sense that swap space is twice as big as | > memory? | | swap space doesn't need to be twice as big as ram. That's fixed long | ago. | | swap+ram is the total amount of virtual memory that you can use in | vmware. | | > And the swap partition has limit as 2G. So we need to setup 8 swap | > partitions if we want 16G swap. | | that's a silly restriction of mkswap, the kernel doesn't care, it can | handle way more than 2G (however there's an high bound at some | unpractical level, to go safe the math limit should be re-encoded in | mkswap, of course it changes for every arch because the pte layout is | different).
Heh, you hit one of my personal todo list items (larger swap spaces :), so I'll be looking into it, or trying to help anyone else on it if they want it.
Signature · 1 line
~Randy
Quoted 10 lines · Randy.Dunlap
On Mon, 28 Oct 2002, Andrea Arcangeli wrote:
| that's a silly restriction of mkswap, the kernel doesn't care, it can | handle way more than 2G (however there's an high bound at some | unpractical level, to go safe the math limit should be re-encoded in | mkswap, of course it changes for every arch because the pte layout is | different).Heh, you hit one of my personal todo list items (larger swap spaces :), so I'll be looking into it, or trying to help anyone else on it if they want it.
If you start playing with the swap code, could you please change the on-disk swap struct definition to look like:
union swap_header {
:
:
struct {
char bootbits[1024]; /* Space for disklabel etc. */
unsigned __u32 version;
unsigned __u32 last_page;
unsigned __u32 nr_badpages;
char volume_label[16];
unsigned __u32 padding[121];
unsigned __u32 badpages[1];
} info;
};1) change all of the "int" definitions in info to be __u32, because this is written to disk and we want the sizes to be unambiguous
2) the volume label field has been previously discussed and doesn't impose any compatibility, but allows one to "swapon by label" (old patch URL at http://user.it.uu.se/~mikpe/linux/swap-label/)
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/
Hi Andrea,
Quoted 15 lines · Andrea Arcangeli
Quoted 8 lines
That is exactly the case for vmware ram file. VMware only use it to share memory. Those are the virtual machine's memory. We don't want to write it back to disk and we don't care what is left on the file system because when vmware exit, we will throw the guest ram data away just like a real machine power off ram will lost. We are not talking about machine using flash ram :-).
Quoted 6 lines
It is kswapd try to flush the data and it should take response to handle the error. If it fail, one thing it should do is keep the page dirty if write back fail. At least not corrupt memory like that.
If we can deliver the error to user program that would be a plus. But this need to be fix frist.
Quoted 8 lines
as said this cannot be fixed easily in kernel, or it would be trivial to lockup a machine by filling the fs changing the
i_sizeof a file and by marking all ram in the machine dirty in the hole, the vm must be allowed to discard those pages and invaliding those posted writes. At least until a true solution will be available you should change vmware to preallocate the file, then it will work fine because you will catch the ENOSPC error during the preallocation. If you work on shmfs that will be very quick indeed.
I still think throwing process's page away if write fail is bad. If kernel drop the data, that process is not able to run correctly anyway. Why not keep the page here and let oom killer to pick up the process to kill. In this way, we at least have some process able to run correctly instead of every process which hit the out of disk space has some bad data.
Cheers
Chris
Quoted 27 lines · On Mon, Oct 28, 2002 at 11:58:31AM -0800, [email protected] wrote:
Hi Andrea,
Quoted 15 lines · Andrea Arcangeli
Quoted 8 lines
That is exactly the case for vmware ram file. VMware only use it to share memory. Those are the virtual machine's memory. We don't want to write it back to disk and we don't care what is left on the file system because when vmware exit, we will throw the guest ram data away just like a real machine power off ram will lost. We are not talking about machine using flash ram :-).
Quoted 6 lines
It is kswapd try to flush the data and it should take response to handle the error. If it fail, one thing it should do is keep the page dirty if write back fail. At least not corrupt memory like that.
If we can deliver the error to user program that would be a plus. But this need to be fix frist.
Quoted 8 lines
as said this cannot be fixed easily in kernel, or it would be trivial to lockup a machine by filling the fs changing the
i_sizeof a file and by marking all ram in the machine dirty in the hole, the vm must be allowed to discard those pages and invaliding those posted writes. At least until a true solution will be available you should change vmware to preallocate the file, then it will work fine because you will catch the ENOSPC error during the preallocation. If you work on shmfs that will be very quick indeed.
Quoted 6 lines
I still think throwing process's page away if write fail is bad. If kernel drop the data, that process is not able to run correctly anyway. Why not keep the page here and let oom killer to pick up the process to kill. In this way, we at least have some process able to run correctly instead of every process which hit the out of disk space has some bad data.
the reason it isn't easily feasible is that you can learn that the writepage fails way after the process isn't mapping the page anymore and we can't keep unowned unwriteable dirty pages around for long, we've to drop them. if the vm tried to writepage it means no one single task had such page mapped, so at that time of the failure we've no clue of who to notify for such page, all tasks just thought to have written the data successfully when the pte dirty bit is been trasmitted to the page dirty bit, by the time the page dirty bit is lost because writepage fails it's too late to let know the task about it, the task may be exited long ago. We lose track of the task when we trasmit the dirty information from pagetable to pte, and only after that happens we will attempt a writepage. So as far as I can tell the only way to fix it and to for example reliably send a signal to a task to notify a write is been lost, is to run the fs get_block(create = 1) while trasmitting the dirty bit from pte_t to page_t, which isn't a trivial change, certainly not something that would be confortable to do in 2.4 and it would affect performance, it is much cleaner and efficient to deal with the fs only at the page_t phyical pagecache layer rather than at the pte layer. Lefting unowned, unfreeable pages around by marking the page dirty when block_write_full_page fails, doesn't look a viable option, the kernel could do nothing but loop forever trying to write in such case.
Andrea
Quoted 2 lines · Andrea Arcangeli
the reason it isn't easily feasible is that you can learn that the writepage fails way after the process isn't mapping the page anymore and
Hmm, nice to know that. When I do bigmm test, I see it kswapd call to writepage. You are telling me at that point, bigmm is dead already? I did not know enough about the vm system. If program did not map this memory any more. It is ok to drop then. VMware do not care about what is left on the ram file at all.
Thanks for you long explain.
Chris
Quoted 21 lines
we can't keep unowned unwriteable dirty pages around for long, we've to drop them. if the vm tried to writepage it means no one single task had such page mapped, so at that time of the failure we've no clue of who to notify for such page, all tasks just thought to have written the data successfully when the pte dirty bit is been trasmitted to the page dirty bit, by the time the page dirty bit is lost because writepage fails it's too late to let know the task about it, the task may be exited long ago. We lose track of the task when we trasmit the dirty information from pagetable to pte, and only after that happens we will attempt a writepage. So as far as I can tell the only way to fix it and to for example reliably send a signal to a task to notify a write is been lost, is to run the fs
get_block(create = 1)while trasmitting the dirty bit frompte_ttopage_t, which isn't a trivial change, certainly not something that would be confortable to do in 2.4 and it would affect performance, it is much cleaner and efficient to deal with the fs only at thepage_tphyical pagecache layer rather than at the pte layer. Lefting unowned, unfreeable pages around by marking the page dirty whenblock_write_full_pagefails, doesn't look a viable option, the kernel could do nothing but loop forever trying to write in such case.Andrea