Deadlock during heavy write activity to userspace NFS server on local NFS mount

PATCH In review Avi Kivity 6 participants 22 messages Jul 26, 2004 – Jul 29, 2004

Share & tools

Avi Kivity

author

2004-07-26 13:11 UTC #0 lore raw

PATCH 1/1 Deadlock during heavy write activity to userspace NFS server on local NFS mount

On heavy write activity, allocators wait synchronously for kswapd to free some memory. But if kswapd is freeing memory via a userspace NFS server, that server could be waiting for kswapd, and the system seizes instantly.

This patch (against RHEL 2.4.21-15EL, but should apply either literally or conceptually to other kernels) allows a process to declare itself as kswapd's little helper, and thus will not have to wait on kswapd.

include/linux/prctl.h +22 -1

@@ -43,5 +43,10 @@
 # define PR_TIMING_TIMESTAMP	1	/* Accurate timestamp based
 						   process timing */
 
+/* Get/set PF_MEMALLOC task flag bit */
+#define PR_GET_KSWAPD_HELPER 15
+#define PR_SET_KSWAPD_HELPER 16
+
+
 #endif /* _LINUX_PRCTL_H */
--- a/kernel/sys.c	2003-10-23 09:00:00.000000000 +0200
+++ b/kernel/sys.c	2004-07-21 13:42:59.000000000 +0300

@@ -1400,6 +1400,22 @@ asmlinkage long sys_prctl(int option, un
 			}
 			current->keep_capabilities = arg2;
 			break;
+		case PR_GET_KSWAPD_HELPER:
+			if (current->flags & PF_MEMALLOC)
+				error = 1;
+			break;
+		case PR_SET_KSWAPD_HELPER:
+			switch (arg2) {
+				case 0:
+					current->flags &= ~PF_MEMALLOC;
+					break;
+				case 1:
+					current->flags |= PF_MEMALLOC;
+					break;
+				default:
+					error = -EINVAL;
+			}
+			break;
 		default:
 			error = -EINVAL;
 			break;

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Pavel Machek

2004-07-27 20:10 UTC #1 lore raw

Hi!

Quoted 10 lines

On heavy write activity, allocators wait synchronously for kswapd to free some memory. But if kswapd is freeing memory via a userspace NFS server, that server could be waiting for kswapd, and the system seizes instantly.
This patch (against RHEL 2.4.21-15EL, but should apply either literally
or conceptually to other kernels) allows a process to declare itself as
kswapd's little helper, and thus will not have to wait on kswapd.

Ok, but what if its memory runs out, anyway?

Pavel

Signature · 1 line

64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms

Avi Kivity

author

2004-07-27 20:23 UTC #2 lore raw

Quoted 5 lines · Pavel Machek

Quoted 5 lines
On heavy write activity, allocators wait synchronously for kswapd to free some memory. But if kswapd is freeing memory via a userspace NFS server, that server could be waiting for kswapd, and the system seizes instantly.

Quoted 10 lines

Quoted 5 lines
This patch (against RHEL 2.4.21-15EL, but should apply either literally or conceptually to other kernels) allows a process to declare itself as kswapd's little helper, and thus will not have to wait on kswapd.
Ok, but what if its memory runs out, anyway?

Tough. What if kswapd's memory runs out?

A more complete solution would be to assign memory reserve levels below which a process starts allocating synchronously. For example, normal processes must have >20MB to make forward progress, kswapd wants >15MB and the NFS server needs >10MB. Some way would be needed to express the dependencies.

I think more and more people will hit this problem as filesystems become more complex due to clustering, and migrate to userspace where it can be more easily managed.

Avi

Pavel Machek

2004-07-27 20:34 UTC #3 lore raw

Hi!

Quoted 13 lines

Quoted 13 lines
Quoted 12 lines
On heavy write activity, allocators wait synchronously for kswapd to free some memory. But if kswapd is freeing memory via a userspace NFS server, that server could be waiting for kswapd, and the system seizes instantly.
This patch (against RHEL 2.4.21-15EL, but should apply either literally
or conceptually to other kernels) allows a process to declare itself as
kswapd's little helper, and thus will not have to wait on kswapd.

Quoted 5 lines

Quoted 4 lines
Ok, but what if its memory runs out, anyway?
Tough. What if kswapd's memory runs out?

I'd hope that kswapd was carefully to make sure that it always has enough pages...

...it is harder to do the same auditing with userland program.

Quoted 5 lines

A more complete solution would be to assign memory reserve levels below which a process starts allocating synchronously. For example, normal processes must have >20MB to make forward progress, kswapd wants >15MB and the NFS server needs >10MB. Some way would be needed to express the dependencies.

Yes, something like that would be neccessary. I believe it would be slightly more complicated, like

"NFS server needs > 10MB *and working kswapd*", so you'd need 25MB in fact... and this info should be stored in some readable form so that it can be checked.

Pavel

Signature · 2 lines

People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Avi Kivity

author

2004-07-27 21:02 UTC #4 lore raw

Quoted 7 lines · Pavel Machek

I'd hope that kswapd was carefully to make sure that it always has enough pages...
...it is harder to do the same auditing with userland program.

Very true. But is a kernel thread like kswapd depends on a userspace program, then that program better be well behaved.

Quoted 4 lines

Quoted 3 lines
A more complete solution would be to assign memory reserve levels below which a process starts allocating synchronously. For example, normal processes must have >20MB to make forward progress, kswapd wants >15MB and the NFS server needs >10MB. Some way would be needed to express the dependencies.

Quoted 9 lines

Yes, something like that would be neccessary. I believe it would be slightly more complicated, like
"NFS server needs > 10MB *and working kswapd*", so you'd need 25MB in fact... and this info should be stored in some readable form so that it can be checked.

If the NFS server needed kswapd, we'd deadlock pretty soon, as kswapd *really* needs the NFS server. In our case, all block I/O is done using unbuffered I/O, and all memory is preallocated, so we don't need kswapd at all, just that small bit of memory that syscalls consume.

If the NFS server really needs kswapd, then there'd better be two of them. Regular processes would depend on one kswapd, which depends on the NFS server, which depends on the second kswapd, which depends on the hardware alone. It should be fun trying to describe that topology to the kernel through some API.

Our filesystem actually does something like that internally, except the dependency chain length is seven, not two.

Avi

Nick Piggin

2004-07-28 01:31 UTC #5 lore raw

Quoted 24 lines · Avi Kivity

Quoted 7 lines · Pavel Machek
I'd hope that kswapd was carefully to make sure that it always has enough pages...
...it is harder to do the same auditing with userland program.
Very true. But is a kernel thread like kswapd depends on a userspace program, then that program better be well behaved.
Quoted 4 lines
Quoted 2 lines
A more complete solution would be to assign memory reserve levels below which a process starts allocating synchronously. For example, normal processes must have >20MB to make forward progress, kswapd wants >15MB and the NFS server needs >10MB. Some way would be needed to express the dependencies.
Quoted 9 lines
Yes, something like that would be neccessary. I believe it would be slightly more complicated, like
"NFS server needs > 10MB *and working kswapd*", so you'd need 25MB in fact... and this info should be stored in some readable form so that it can be checked.

Quoted 6 lines

If the NFS server needed kswapd, we'd deadlock pretty soon, as kswapd *really* needs the NFS server. In our case, all block I/O is done using unbuffered I/O, and all memory is preallocated, so we don't need kswapd at all, just that small bit of memory that syscalls consume.
If the NFS server really needs kswapd, then there'd better be two of them. Regular processes would depend on one kswapd, which depends on the NFS server, which depends on the second kswapd, which depends on the hardware alone. It should be fun trying to describe that topology to the kernel through some API.
Our filesystem actually does something like that internally, except the dependency chain length is seven, not two.

There is some need arising for a call to set the PF_MEMALLOC flag for userspace tasks, so you could probably get a patch accepted. Don't call it KSWAPD_HELPER though, maybe MEMFREE or RECLAIM or RECLAIM_HELPER.

But why is your NFS server needed to reclaim memory? Do you have the filesystem mounted locally?

Trond Myklebust

2004-07-28 02:17 UTC #6 lore raw

På ty , 27/07/2004 klokka 21:29, skreiv Nick Piggin:

Quoted 6 lines

There is some need arising for a call to set the PF_MEMALLOC flag for userspace tasks, so you could probably get a patch accepted. Don't call it KSWAPD_HELPER though, maybe MEMFREE or RECLAIM or RECLAIM_HELPER.
But why is your NFS server needed to reclaim memory? Do you have the filesystem mounted locally?

...and why can't this problem be fixed by judicious use of mlock()?

Cheers,
Trond

Avi Kivity

author

2004-07-28 05:13 UTC #7 lore raw

Quoted 8 lines · Trond Myklebust

På ty , 27/07/2004 klokka 21:29, skreiv Nick Piggin:
Quoted 4 lines
There is some need arising for a call to set the PF_MEMALLOC flag for userspace tasks, so you could probably get a patch accepted. Don't call it KSWAPD_HELPER though, maybe MEMFREE or RECLAIM or RECLAIM_HELPER.

Quoted 9 lines

Quoted 4 lines
But why is your NFS server needed to reclaim memory? Do you have the filesystem mounted locally?
...and why can't this problem be fixed by judicious use of mlock()?

I mlock(MLOCK_ALL) as soon as I wake up in the morning (and I never ask the kernel for more memory), but the kernel likes to allocate memory when performing some syscalls for me.

Avi

Avi Kivity

author

2004-07-28 05:11 UTC #8 lore raw

In reply to Nick Piggin

Quoted 4 lines · Nick Piggin

There is some need arising for a call to set the PF_MEMALLOC flag for userspace tasks, so you could probably get a patch accepted. Don't call it KSWAPD_HELPER though, maybe MEMFREE or RECLAIM or RECLAIM_HELPER.

I don't think my patch is general enough, it deals with only one level of dependencies, and doesn't work if the NFS server (or other process that kswapd depends on) depends on kswapd itself. It was intended more as an RFC than a request for inclusion.

It's probably fine for those with the exact same problem as us.

Quoted 3 lines

But why is your NFS server needed to reclaim memory? Do you have the filesystem mounted locally?

Yes, for use by protocol adapters like samba.

Avi

Nick Piggin

2004-07-28 06:04 UTC #9 lore raw

Quoted 12 lines · Avi Kivity

Quoted 4 lines · Nick Piggin
There is some need arising for a call to set the PF_MEMALLOC flag for userspace tasks, so you could probably get a patch accepted. Don't call it KSWAPD_HELPER though, maybe MEMFREE or RECLAIM or RECLAIM_HELPER.
I don't think my patch is general enough, it deals with only one level of dependencies, and doesn't work if the NFS server (or other process that kswapd depends on) depends on kswapd itself. It was intended more as an RFC than a request for inclusion.
It's probably fine for those with the exact same problem as us.

Well it isn't that you depend on kswapd, but that your task gets called into via page reclaim (to facilitate page reclaim). In which case having the task block in memory allocation can cause a deadlock.

The solution is that PF_MEMALLOC tasks are allowed to access the reserve pool. Dependencies don't matter to this system. It would be your job to ensure all tasks that might need to allocate memory in order to free memory have the flag set.

Quoted 7 lines

Quoted 3 lines
But why is your NFS server needed to reclaim memory? Do you have the filesystem mounted locally?
Yes, for use by protocol adapters like samba.

OK.

Avi Kivity

author

2004-07-28 07:05 UTC #10 lore raw

Quoted 11 lines · Nick Piggin

Quoted 9 lines · Avi Kivity
Quoted 4 lines · Nick Piggin
There is some need arising for a call to set the PF_MEMALLOC flag for userspace tasks, so you could probably get a patch accepted. Don't call it KSWAPD_HELPER though, maybe MEMFREE or RECLAIM or RECLAIM_HELPER.

Quoted 8 lines

Quoted 4 lines
I don't think my patch is general enough, it deals with only one level of dependencies, and doesn't work if the NFS server (or other process that kswapd depends on) depends on kswapd itself. It was intended more as an RFC than a request for inclusion.
It's probably fine for those with the exact same problem as us.
Well it isn't that you depend on kswapd, but that your task gets called into via page reclaim (to facilitate page reclaim). In which case having the task block in memory allocation can cause a deadlock.

In my particular case that's true, so I only depended on kswapd as a side effect of the memory allocation logic. Setting PF_MEMALLOC fixed that.

Quoted 5 lines

The solution is that PF_MEMALLOC tasks are allowed to access the reserve pool. Dependencies don't matter to this system. It would be your job to ensure all tasks that might need to allocate memory in order to free memory have the flag set.

In the general case that's not sufficient. What if the NFS server wrote to ext3 via the VFS? We might have a ton of ext3 pagecache waiting for kswapd to reclaim NFS memory, while kswapd is waiting on the NFS server writing to ext3.

The patch I posted is simple and quite sufficient for my needs, but I'm sure more convoluted cases will turn up where something more complex is needed. Probably one can construct such cases out of in-kernel components like the loop device, dm, and the NFS client and server.

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Nick Piggin

2004-07-28 08:01 UTC #11 lore raw

Quoted 11 lines · Avi Kivity

Quoted 5 lines · Nick Piggin
The solution is that PF_MEMALLOC tasks are allowed to access the reserve pool. Dependencies don't matter to this system. It would be your job to ensure all tasks that might need to allocate memory in order to free memory have the flag set.
In the general case that's not sufficient. What if the NFS server wrote to ext3 via the VFS? We might have a ton of ext3 pagecache waiting for kswapd to reclaim NFS memory, while kswapd is waiting on the NFS server writing to ext3.

It is sufficient.

You didn't explain your example very well, but I'll assume it is the following:

dirty NFS data -> NFS server on localhost -> ext3 filesystem.

So kswapd tries to reclaim some memory and writes out the dirty NFS data. The NFS server then writes this data to ext3 (it can do this because it is PF_MEMALLOC). The data gets written out, the NFS server tells the client it is clean, kswapd continues.

Right?

Avi Kivity

author

2004-07-28 07:57 UTC #12 lore raw

Quoted 16 lines · Nick Piggin

Quoted 4 lines · Avi Kivity
Quoted 1 line · Nick Piggin
Quoted 9 lines
Quoted 4 lines
The solution is that PF_MEMALLOC tasks are allowed to access the reserve pool. Dependencies don't matter to this system. It would be your job to ensure all tasks that might need to allocate memory in order to free memory have the flag set.
In the general case that's not sufficient. What if the NFS server wrote to ext3 via the VFS? We might have a ton of ext3 pagecache waiting for kswapd to reclaim NFS memory, while kswapd is waiting on the NFS server writing to ext3.

Quoted 6 lines

It is sufficient.
You didn't explain your example very well, but I'll assume it is the following:
dirty NFS data -> NFS server on localhost -> ext3 filesystem.

That's what I meant, sorry for not making it clear.

Quoted 8 lines

So kswapd tries to reclaim some memory and writes out the dirty NFS data. The NFS server then writes this data to ext3 (it can do this because it is PF_MEMALLOC). The data gets written out, the NFS server tells the client it is clean, kswapd continues.
Right?

What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.

An even more complex case is when ext3 depends on some other process, say it is mounted on a loopback nbd.

dirty NFS data -> NFS server -> ext3 -> nbd -> nbd server on localhost
-> ext3/raw device

You can't have both the NFS server and the nbd server PF_MEMALLOC, since the NFS server may consume all memory, then wait for the nbd server to reclaim.

The solution I have in mind is to replace the sync allocation logic from

   if (free_mem() < some_global_limit && !current->PF_MEMALLOC)
       wait_for_kswapd()

if (free_mem() < current->limit)
wait_for_kswapd()

kswapd would have the lowest ->limit, other processes as their place in the food chain dictates.

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Nick Piggin

2004-07-28 09:09 UTC #13 lore raw

Quoted 37 lines · Avi Kivity

Quoted 18 lines · Nick Piggin
Quoted 13 lines · Avi Kivity
Quoted 5 lines · Nick Piggin
The solution is that PF_MEMALLOC tasks are allowed to access the reserve pool. Dependencies don't matter to this system. It would be your job to ensure all tasks that might need to allocate memory in order to free memory have the flag set.
Quoted 2 lines
In the general case that's not sufficient. What if the NFS server wrote to ext3 via the VFS? We might have a ton of ext3 pagecache waiting for kswapd to reclaim NFS memory, while kswapd is waiting on the NFS server writing to ext3.
Quoted 6 lines
It is sufficient.
You didn't explain your example very well, but I'll assume it is the following:
dirty NFS data -> NFS server on localhost -> ext3 filesystem.
That's what I meant, sorry for not making it clear.
Quoted 7 lines
So kswapd tries to reclaim some memory and writes out the dirty NFS data. The NFS server then writes this data to ext3 (it can do this because it is PF_MEMALLOC). The data gets written out, the NFS server tells the client it is clean, kswapd continues.

Quoted 5 lines

Quoted 1 line
Right?
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.

The NFS server should do the writeout a page at a time.

Quoted 7 lines

An even more complex case is when ext3 depends on some other process, say it is mounted on a loopback nbd.
dirty NFS data -> NFS server -> ext3 -> nbd -> nbd server on localhost
-> ext3/raw device
You can't have both the NFS server and the nbd server PF_MEMALLOC, since the NFS server may consume all memory, then wait for the nbd server to reclaim.

The memory allocators will block when memory reaches the reserved mark. Page reclaim will ask NFS to free one page, so the server will write something out to the filesystem, this will cause the nbd server (also PF_MEMALLOC) to write out to its backing filesystem.

Quoted 11 lines

The solution I have in mind is to replace the sync allocation logic from
   if (free_mem() < some_global_limit && !current->PF_MEMALLOC)
       wait_for_kswapd()
to
   if (free_mem() < current->limit)
       wait_for_kswapd()
kswapd would have the lowest ->limit, other processes as their place in the food chain dictates.

I think this is barking up the wrong tree. It really doesn't matter what process is freeing memory. There isn't really anything special about the way kswapd frees memory.

Avi Kivity

author

2004-07-28 10:11 UTC #14 lore raw

Quoted 24 lines · Nick Piggin

Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
The solution is that PF_MEMALLOC tasks are allowed to access the reserve pool. Dependencies don't matter to this system. It would be your job to ensure all tasks that might need to allocate memory in order to free memory have the flag set.
Quoted 8 lines
Quoted 2 lines
In the general case that's not sufficient. What if the NFS server wrote to ext3 via the VFS? We might have a ton of ext3 pagecache waiting for kswapd to reclaim NFS memory, while kswapd is waiting on the NFS server writing to ext3.
It is sufficient.
You didn't explain your example very well, but I'll assume it is the following:
dirty NFS data -> NFS server on localhost -> ext3 filesystem.
That's what I meant, sorry for not making it clear.
Quoted 5 lines
So kswapd tries to reclaim some memory and writes out the dirty NFS data. The NFS server then writes this data to ext3 (it can do this because it is PF_MEMALLOC). The data gets written out, the NFS server tells the client it is clean, kswapd continues.

Quoted 6 lines

Quoted 5 lines
Quoted 1 line
Right?
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.
The NFS server should do the writeout a page at a time.

The NFS server writes not only in response to page reclaim (as a local NFS client), but also in response to pressure from non-local clients. If both ext3 and NFS have the same allocation limits, NFS may starve out ext3.

(In my case the NFS server actually writes data asynchronously, so it doesn't really know it is responding to page reclaim, but the problem occurs even in a synchrounous NFS server.)

Quoted 6 lines

Quoted 6 lines
An even more complex case is when ext3 depends on some other process, say it is mounted on a loopback nbd.
dirty NFS data -> NFS server -> ext3 -> nbd -> nbd server on
localhost -> ext3/raw device

Quoted 6 lines

Quoted 2 lines
You can't have both the NFS server and the nbd server PF_MEMALLOC, since the NFS server may consume all memory, then wait for the nbd server to reclaim.
The memory allocators will block when memory reaches the reserved mark. Page reclaim will ask NFS to free one page, so the server will write something out to the filesystem, this will cause the nbd server (also PF_MEMALLOC) to write out to its backing filesystem.

If NFS and nbd have the same limit, then NFS may cause nbd to stall. We've already established that NFS must be PF_MEMALLOC, so nbd must be PF_MEMALLOC_HARDER or something like that.

Quoted 11 lines

The solution I have in mind is to replace the sync allocation logic from
Quoted 9 lines
   if (free_mem() < some_global_limit && !current->PF_MEMALLOC)
       wait_for_kswapd()
to
   if (free_mem() < current->limit)
       wait_for_kswapd()

Quoted 6 lines

Quoted 1 line
kswapd would have the lowest ->limit, other processes as their place in the food chain dictates.
I think this is barking up the wrong tree. It really doesn't matter what process is freeing memory. There isn't really anything special about the way kswapd frees memory.

To free memory you need (a) to allocate memory (b) possibly wait for some freeing process to make some progress. That means all processes in the freeing chain must be able to allocate at least some memory. If two processes in the chain share the same blocking logic, they may deadlock on each other.

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Nick Piggin

2004-07-28 10:31 UTC #15 lore raw

Quoted 9 lines · Avi Kivity

Quoted 3 lines · Nick Piggin
Quoted 2 lines
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.
The NFS server should do the writeout a page at a time.
The NFS server writes not only in response to page reclaim (as a local NFS client), but also in response to pressure from non-local clients. If both ext3 and NFS have the same allocation limits, NFS may starve out ext3.

What do you mean starve out ext3? ext3 gets written to *by the NFS server* which is PF_MEMALLOC.

Quoted 2 lines

(In my case the NFS server actually writes data asynchronously, so it doesn't really know it is responding to page reclaim, but the problem occurs even in a synchrounous NFS server.)

I can't see this being the responsibility of the kernel. The NFS server could probably find out if it is servicing a loopback request or not. Remote requests don't help to free memory... unless maybe you want a filesystem on a remote nbd to be exported back to server via NFS or something crazy.

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
An even more complex case is when ext3 depends on some other process, say it is mounted on a loopback nbd.
dirty NFS data -> NFS server -> ext3 -> nbd -> nbd server on
localhost -> ext3/raw device
You can't have both the NFS server and the nbd server PF_MEMALLOC, since the NFS server may consume all memory, then wait for the nbd server to reclaim.

Quoted 7 lines

Quoted 4 lines
The memory allocators will block when memory reaches the reserved mark. Page reclaim will ask NFS to free one page, so the server will write something out to the filesystem, this will cause the nbd server (also PF_MEMALLOC) to write out to its backing filesystem.
If NFS and nbd have the same limit, then NFS may cause nbd to stall. We've already established that NFS must be PF_MEMALLOC, so nbd must be PF_MEMALLOC_HARDER or something like that.

No, your NFS server has to be coded differently. You can't allow it to use up all PF_MEMALLOC memory just because it can.

Quoted 16 lines

Quoted 16 lines
The solution I have in mind is to replace the sync allocation logic from
Quoted 10 lines
   if (free_mem() < some_global_limit && !current->PF_MEMALLOC)
       wait_for_kswapd()
to
   if (free_mem() < current->limit)
       wait_for_kswapd()
kswapd would have the lowest ->limit, other processes as their place in the food chain dictates.

Quoted 7 lines

Quoted 3 lines
I think this is barking up the wrong tree. It really doesn't matter what process is freeing memory. There isn't really anything special about the way kswapd frees memory.
To free memory you need (a) to allocate memory (b) possibly wait for some freeing process to make some progress. That means all processes in the freeing chain must be able to allocate at least some memory. If two processes in the chain share the same blocking logic, they may deadlock on each other.

The PF_MEMALLOC path isn't to be used like that. If a *single* PF_MEMALLOC task were to allocate all its memory then that would be a bug too.

Avi Kivity

author

2004-07-28 11:48 UTC #16 lore raw

Quoted 11 lines · Nick Piggin

Quoted 9 lines · Avi Kivity
Quoted 3 lines · Nick Piggin
Quoted 2 lines
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.
The NFS server should do the writeout a page at a time.

Quoted 5 lines

Quoted 2 lines
The NFS server writes not only in response to page reclaim (as a local NFS client), but also in response to pressure from non-local clients. If both ext3 and NFS have the same allocation limits, NFS may starve out ext3.
What do you mean starve out ext3? ext3 gets written to *by the NFS server* which is PF_MEMALLOC.

When the NFS server writes, it allocates pagecache and temporary objects. When ext3 writes, it allocates temporary objects. If the NFS server writes too much, ext3 can't allocate memory, and will never be able to allocate memory.

Quoted 7 lines

Quoted 4 lines
(In my case the NFS server actually writes data asynchronously, so it doesn't really know it is responding to page reclaim, but the problem occurs even in a synchrounous NFS server.)
I can't see this being the responsibility of the kernel. The NFS server could probably find out if it is servicing a loopback request or not.

It might be emptying its own caches. Or any one of a hundred other things. Everything can be hacked around (as I did with PF_MEMALLOC, and relying on my server to write directly), but that's not a general solution.

Quoted 4 lines

Remote requests don't help to free memory... unless maybe you want a filesystem on a remote nbd to be exported back to server via NFS or something crazy.

Cycles can easily happen in clusters, especially with failover and failback.

Quoted 13 lines

Quoted 13 lines
Quoted 11 lines
Quoted 7 lines
An even more complex case is when ext3 depends on some other process, say it is mounted on a loopback nbd.
dirty NFS data -> NFS server -> ext3 -> nbd -> nbd server on
localhost -> ext3/raw device
You can't have both the NFS server and the nbd server PF_MEMALLOC, since the NFS server may consume all memory, then wait for the nbd server to reclaim.
The memory allocators will block when memory reaches the reserved mark. Page reclaim will ask NFS to free one page, so the server will write something out to the filesystem, this will cause the nbd server (also PF_MEMALLOC) to write out to its backing filesystem.

Quoted 5 lines

Quoted 1 line
If NFS and nbd have the same limit, then NFS may cause nbd to stall. We've already established that NFS must be PF_MEMALLOC, so nbd must be PF_MEMALLOC_HARDER or something like that.
No, your NFS server has to be coded differently. You can't allow it to use up all PF_MEMALLOC memory just because it can.

There is no API that the NFS server can use to block and unblock on memory allocation.

My proposal extends memory allocation in general to block and unblock in a way that prevents deadlocks, providing the reservation levels have been set up correctly. And it can work cluster-wide.

Quoted 19 lines

Quoted 19 lines
Quoted 17 lines
The solution I have in mind is to replace the sync allocation logic from
Quoted 10 lines
   if (free_mem() < some_global_limit && !current->PF_MEMALLOC)
       wait_for_kswapd()
to
   if (free_mem() < current->limit)
       wait_for_kswapd()
kswapd would have the lowest ->limit, other processes as their place in the food chain dictates.
I think this is barking up the wrong tree. It really doesn't matter what process is freeing memory. There isn't really anything special about the way kswapd frees memory.

Quoted 6 lines

Quoted 2 lines
To free memory you need (a) to allocate memory (b) possibly wait for some freeing process to make some progress. That means all processes in the freeing chain must be able to allocate at least some memory. If two processes in the chain share the same blocking logic, they may deadlock on each other.
The PF_MEMALLOC path isn't to be used like that. If a *single* PF_MEMALLOC task were to allocate all its memory then that would be a bug too.

Again, there is no way for a task to know how much memory to allocate. Even if it writes out one page at a time, if it's PF_MEMALLOC and it generates dirty pagecache, it will soon consume all memory. Would you have the server sleep() after every write?

Note that my proposal is a generalization of PF_MEMALLOC; instead of setting the PF_MEMALLOC flag in kswapd you can set current->min_free_mem_to_allocate_nonblockingly to 0.

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Nick Piggin

2004-07-29 08:29 UTC #17 lore raw

Quoted 15 lines · Avi Kivity

Quoted 13 lines · Nick Piggin
Quoted 11 lines · Avi Kivity
Quoted 3 lines · Nick Piggin
Quoted 2 lines
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.
The NFS server should do the writeout a page at a time.

Quoted 9 lines

Quoted 5 lines
Quoted 2 lines
The NFS server writes not only in response to page reclaim (as a local NFS client), but also in response to pressure from non-local clients. If both ext3 and NFS have the same allocation limits, NFS may starve out ext3.
What do you mean starve out ext3? ext3 gets written to *by the NFS server* which is PF_MEMALLOC.
When the NFS server writes, it allocates pagecache and temporary objects. When ext3 writes, it allocates temporary objects. If the NFS server writes too much, ext3 can't allocate memory, and will never be able to allocate memory.

That is because your NFS server shouldn't hog as much memory as it likes when it is PF_MEMALLOC. The entire writeout path should do a page at a time if it is PF_MEMALLOC. Ie, the server should be doing write, fsync.

But now that I think about it, I guess you may not be able to distinguish that from regular writeout, so doing a page at a time would hurt performance too much.

Hmm so I guess the idea of a per task reserve limit may be the way to do it, yes. Thanks for bearing with me!

Marcelo Tosatti

2004-07-29 13:19 UTC #18 lore raw

Quoted 41 lines · Nick Piggin

Quoted 29 lines · Avi Kivity
Quoted 22 lines · Nick Piggin
Quoted 14 lines · Avi Kivity
Quoted 6 lines · Nick Piggin
Quoted 5 lines
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.
The NFS server should do the writeout a page at a time.
Quoted 5 lines
The NFS server writes not only in response to page reclaim (as a local NFS client), but also in response to pressure from non-local clients. If both ext3 and NFS have the same allocation limits, NFS may starve out ext3.
Quoted 3 lines
What do you mean starve out ext3? ext3 gets written to *by the NFS server*
which is PF_MEMALLOC.
Quoted 5 lines
When the NFS server writes, it allocates pagecache and temporary objects. When ext3 writes, it allocates temporary objects. If the NFS server writes too much, ext3 can't allocate memory, and will never be able to allocate memory.
That is because your NFS server shouldn't hog as much memory as it likes when it is PF_MEMALLOC. The entire writeout path should do a page at a time if it is PF_MEMALLOC. Ie, the server should be doing write, fsync.

Quoted 6 lines

But now that I think about it, I guess you may not be able to distinguish that from regular writeout, so doing a page at a time would hurt performance too much.
Hmm so I guess the idea of a per task reserve limit may be the way to do it, yes. Thanks for bearing with me!

Hi,

By reading the discussion I also agree that we need levels of "allowed deepness" into the reservations.

We could have a global limit for normal allocators, and per-task limit for special "kswapd helpers" (which run with PF_MEMALLOC). And with kswapd being the most "deep" eater, as you guys said.

The thing is, those deadlocks are quite uncommon special cases and we will need to change core VM logic to handle them. Well...

We need to come up with a generic way of doing it, as Avi says, otherwise people will have to keep doing "hacks" to make it work.

Someone needs to sit down and come up with a design. It shouldnt be that hard.

Just my two pennies...

Avi Kivity

author

2004-07-29 16:14 UTC #19 lore raw

In reply to Nick Piggin

Quoted 26 lines · Nick Piggin

Quoted 24 lines · Avi Kivity
Quoted 19 lines · Nick Piggin
Quoted 6 lines · Avi Kivity
Quoted 2 lines · Nick Piggin
Quoted 2 lines
What's stopping the NFS server from ooming the machine then? Every time some bit of memory becomes free, the server will consume it instantly. Eventually ext3 will not be able to write anything out because it is out of memory.
Quoted 8 lines
Quoted 1 line
The NFS server should do the writeout a page at a time.
The NFS server writes not only in response to page reclaim (as a local NFS client), but also in response to pressure from non-local clients. If both ext3 and NFS have the same allocation limits, NFS may starve out ext3.
What do you mean starve out ext3? ext3 gets written to *by the NFS server* which is PF_MEMALLOC.

Quoted 7 lines

Quoted 2 lines
When the NFS server writes, it allocates pagecache and temporary objects. When ext3 writes, it allocates temporary objects. If the NFS server writes too much, ext3 can't allocate memory, and will never be able to allocate memory.
That is because your NFS server shouldn't hog as much memory as it likes when it is PF_MEMALLOC. The entire writeout path should do a page at a time if it is PF_MEMALLOC. Ie, the server should be doing write, fsync.

We attempted to use sync local mounts (not what you are suggesting: on the NFS client side, without the PF_MEMALLOC hack) and still got the same deadlock. I am unable to explain why.

Quoted 8 lines

But now that I think about it, I guess you may not be able to distinguish that from regular writeout, so doing a page at a time would hurt performance too much.
Hmm so I guess the idea of a per task reserve limit may be the way to do it, yes. Thanks for bearing with me!

It was my pleasure.

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Mikulas Patocka

2004-07-28 12:08 UTC #20 lore raw

In reply to Pavel Machek

Quoted 32 lines

Hi!
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
On heavy write activity, allocators wait synchronously for kswapd to free some memory. But if kswapd is freeing memory via a userspace NFS server, that server could be waiting for kswapd, and the system seizes instantly.
This patch (against RHEL 2.4.21-15EL, but should apply either literally
or conceptually to other kernels) allows a process to declare itself as
kswapd's little helper, and thus will not have to wait on kswapd.
Quoted 5 lines
Quoted 4 lines
Ok, but what if its memory runs out, anyway?
Tough. What if kswapd's memory runs out?
I'd hope that kswapd was carefully to make sure that it always has enough pages...
...it is harder to do the same auditing with userland program.
Quoted 5 lines
A more complete solution would be to assign memory reserve levels below which a process starts allocating synchronously. For example, normal processes must have >20MB to make forward progress, kswapd wants >15MB and the NFS server needs >10MB. Some way would be needed to express the dependencies.

Quoted 6 lines

Yes, something like that would be neccessary. I believe it would be slightly more complicated, like
"NFS server needs > 10MB *and working kswapd*", so you'd need 25MB in fact... and this info should be stored in some readable form so that it can be checked.

Hi!

And if the NFS server is waiting for some lock that is held by another process that is wating for kswapd...? It won't help.

The solution would be a limit for dirty pages on NFS --- if you say that less than 1/8 of memory might be dirty NFS pages, than you can keep system stable even if NFS writes starve. If you export NFS filesystem via NFSD again, even this woudn't help, but there's no fix for this case.

Mikulas

Avi Kivity

author

2004-07-28 12:18 UTC #21 lore raw

Quoted 12 lines · Mikulas Patocka

Hi!
And if the NFS server is waiting for some lock that is held by another process that is wating for kswapd...? It won't help.
The solution would be a limit for dirty pages on NFS --- if you say that less than 1/8 of memory might be dirty NFS pages, than you can keep system stable even if NFS writes starve. If you export NFS filesystem via NFSD again, even this woudn't help, but there's no fix for this case.

Oh yes there is. You can have different limits for each export, with the nested export having a lower limit.

Say the first export may dirty at most 200MB, and the nested export at most 180MB. So even if there are heavy writes against the nested export, it can always make progress by writing to the outer export, and if the system has more than 200MB of memory, the external export can make progress by writing out to the filesystem.

That is essentially my suggestion regarding reservation levels, but expressed in allocate-at-most terms instead of leave-at-least. And nested NFS is a fine example to show the current problems.

(of course, when nesting NFS one also needs to reserve threads so each export has at least one thread available to service it. but that's another story)

Signature · 1 line

Do not meddle in the internals of kernels, for they are subtle and quick to panic.