problem with nfs latency during high IO
Subsystems (2)
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3 x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.
Unfortunately my clients are running fedora core 14 (x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the file fs/nfs/inode.c is not there and I can't find the rpm that contains it.
As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
Thanks a lot in advance for your help, please let me know if I can provide any more information. j
Hi Judith-
Quoted 9 lines · Judith Flo Gaya
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
Quoted 7 lines
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3
x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.Unfortunately my clients are running fedora core 14 (
x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the filefs/nfs/inode.cis not there and I can't find the rpm that contains it.As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
The problem is that Linux caches writes aggressively. That makes flushing before the GETATTR take a long time in some cases. On some versions of Linux, it could be an indefinite amount of time; recently we added a bit of logic to make the GETATTR code path hold up additional application writes so it would be able to squeeze in the GETATTR to get a consistent snapshot of mtime and size.
Another issue is: what if other clients are writing to the file? Those writes won't be seen on your client, either in the form of data changes or mtime/size updates, until your client's attribute cache times out (or the file is unlocked or closed).
The best you can do for now is to lower the amount of dirty data the client allows to be outstanding, thus reducing the amount of time it takes for a flush to complete. This is done with a sysctl, I believe "vm.dirty_ratio," and affects all file systems on the client. Alternately, the client file system in question can be mounted with "sync" to cause writes to go to the server immediately, but that has other significant performance implications.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hello Chuck,
Quoted 15 lines · Chuck Lever
Hi Judith-
Quoted 11 lines · Judith Flo Gaya
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3
x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.
Quoted 7 lines
Quoted 5 lines
Unfortunately my clients are running fedora core 14 (
x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the filefs/nfs/inode.cis not there and I can't find the rpm that contains it.As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
But it is not like this when I use a RHEL6 as a client to those servers, in this case, the ls only last for some seconds, nothing like the minutes that it takes from my fedora.
Quoted 3 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
BTW i understand the time issue, but again, if the version of the kernel that the red hat has installed allows me to get the information soon, why a newer kernel in fedora does not?
Quoted 2 lines
The problem is that Linux caches writes aggressively. That makes flushing before the GETATTR take a long time in some cases. On some versions of Linux, it could be an indefinite amount of time; recently we added a bit of logic to make the GETATTR code path hold up additional application writes so it would be able to squeeze in the GETATTR to get a consistent snapshot of mtime and size.
I thought that the purpose of the patch was specifically to allow the client to get the stat(2) info faster than before, so that this aggressive behavior doesn't impact so much the performance of the stat petition.
Quoted 2 lines
Another issue is: what if other clients are writing to the file? Those writes won't be seen on your client, either in the form of data changes or mtime/size updates, until your client's attribute cache times out (or the file is unlocked or closed).
I didn't consider it, big issue indeed. Then how is RHEL doing it not to have the problem???
Quoted 2 lines
The best you can do for now is to lower the amount of dirty data the client allows to be outstanding, thus reducing the amount of time it takes for a flush to complete. This is done with a sysctl, I believe "vm.dirty_ratio," and affects all file systems on the client. Alternately, the client file system in question can be mounted with "sync" to cause writes to go to the server immediately, but that has other significant performance implications.
I'll give it a try and let you know how the new tests are doing.
I already considered the sync parameter but of course the performance of the copy lowers to unacceptable times (from 6min to 40min)
Thanks,
j
Quoted 18 lines · Judith Flo Gaya
Hello Chuck,
Quoted 15 lines · Chuck Lever
Hi Judith-
Quoted 11 lines · Judith Flo Gaya
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3
x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Unfortunately my clients are running fedora core 14 (
x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the filefs/nfs/inode.cis not there and I can't find the rpm that contains it.As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
But it is not like this when I use a RHEL6 as a client to those servers, in this case, the ls only last for some seconds, nothing like the minutes that it takes from my fedora.
Which Fedora systems, exactly? The fix I described below is almost certainly in RHEL 6.
Quoted 4 lines
Quoted 3 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
Dirty data is data that your application has written to the file but which hasn't been flushed to the server's disk. This data resides in the client's page cache, on its way to the server.
Quoted 1 line
BTW i understand the time issue, but again, if the version of the kernel that the red hat has installed allows me to get the information soon, why a newer kernel in fedora does not?
Sounds like a bug. Fedora kernels newer than 2.6.32 should work as well as, or better than, RHEL 6.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 35 lines · Chuck Lever
Quoted 18 lines · Judith Flo Gaya
Hello Chuck,
Quoted 15 lines · Chuck Lever
Hi Judith-
Quoted 11 lines · Judith Flo Gaya
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3
x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Unfortunately my clients are running fedora core 14 (
x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the filefs/nfs/inode.cis not there and I can't find the rpm that contains it.As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
But it is not like this when I use a RHEL6 as a client to those servers, in this case, the ls only last for some seconds, nothing like the minutes that it takes from my fedora.
Which Fedora systems, exactly? The fix I described below is almost certainly in RHEL 6.
Quoted 3 lines
Quoted 3 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
Quoted 7 lines
Quoted 1 line
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
Dirty data is data that your application has written to the file but which hasn't been flushed to the server's disk. This data resides in the client's page cache, on its way to the server.
Quoted 1 line
BTW i understand the time issue, but again, if the version of the kernel that the red hat has installed allows me to get the information soon, why a newer kernel in fedora does not?
Sounds like a bug. Fedora kernels newer than 2.6.32 should work as well as, or better than, RHEL 6.
Looks like commit acdc53b2 "NFS: Replace __nfs_write_mapping with sync_inode()" removes the code that holds i_mutex while trying to flush writes before a GETATTR. This means application writes can possibly starve a stat(2) call. Trond, this seems like a regression...?
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hi Chuck & Judith,
Quoted 7 lines · Chuck Lever
Quoted 3 lines · Chuck Lever
Quoted 1 line · Judith Flo Gaya
BTW i understand the time issue, but again, if the version of the
kernel that the red hat has installed allows me to get the information soon, why a newer kernel in fedora does not?
Quoted 1 line
Quoted 8 lines
Quoted 1 line
Sounds like a bug. Fedora kernels newer than 2.6.32 should work as
well as, or better than, RHEL 6.
Looks like commit
acdc53b2"NFS: Replace__nfs_write_mappingwithsync_inode()" removes the code that holdsi_mutexwhile trying to flush writes before a GETATTR. This means application writes can possibly starve a stat(2) call. Trond, this seems like a regression...?
RHEL 6.0 released with RH kernel 2.6.32-71 and it *does* contain the commit acdc53b2 "NFS: Replace __nfs_write_mapping with sync_inode()" backported to 2.6.32. So, I doubt that the reported bad Fedora performance is due to that commit.
cya,
#
Quoted 20 lines · Chuck Lever
Quoted 18 lines · Judith Flo Gaya
Hello Chuck,
Quoted 15 lines · Chuck Lever
Hi Judith-
Quoted 11 lines · Judith Flo Gaya
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3
x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Unfortunately my clients are running fedora core 14 (
x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the filefs/nfs/inode.cis not there and I can't find the rpm that contains it.As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
But it is not like this when I use a RHEL6 as a client to those servers, in this case, the ls only last for some seconds, nothing like the minutes that it takes from my fedora.
Which Fedora systems, exactly? The fix I described below is almost certainly in RHEL 6.
Fedora Core 14, 64 bit, 2.6.35.6-45
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
Dirty data is data that your application has written to the file but which hasn't been flushed to the server's disk. This data resides in the client's page cache, on its way to the server.
ok, understood. Then the sysctl change that you suggest, I've been checking both distributions, RHEL6 and FC14 and they share the same value... I assume by this that changing this value will not "help", am I right?
Quoted 2 lines
Quoted 1 line
BTW i understand the time issue, but again, if the version of the kernel that the red hat has installed allows me to get the information soon, why a newer kernel in fedora does not?
Sounds like a bug. Fedora kernels newer than 2.6.32 should work as well as, or better than, RHEL 6.
I thought the same, but the test doesn't suggest that this is true ;(
I saw your next message about the difference in the code and that would make a lot of sense!
Thanks for your help!
j
Quoted 25 lines · Judith Flo Gaya
Quoted 22 lines · Chuck Lever
Quoted 20 lines · Judith Flo Gaya
Hello Chuck,
Quoted 17 lines · Chuck Lever
Hi Judith-
Quoted 13 lines · Judith Flo Gaya
Hello,
I was told some days ago that my problem with my NFS system is related to this bug, as the problem that I'm experiencing is quite similar.
The bug : https://bugzilla.redhat.com/show_bug.cgi?id=469848
The link itself explains quite well my issue, I'm just truing to copy a big file (36Gb) to my nfs server and when I try to get an ls -l command to the same folder where I'm copying data, the command gets stuck for some time. This amount of time changes from a few secs to SOME minutes (9' is the current record). I can live with some seconds of delay, but minutes is something quite unacceptable.
As this is an nfs server running on a red hat system (an HP ibrix x9300 with Red Hat 5.3
x86_64, kernel 2.6.18-128) I was told to apply the patch suggested from the bug on my clients.Unfortunately my clients are running fedora core 14 (
x86_64, kernel 2.6.35.6-45) and I can't find the file that they are referring to, the filefs/nfs/inode.cis not there and I can't find the rpm that contains it.
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
As the bug is a very very old one, I took it for granted that is already applied to fedora, but I wanted to make sure that it is looking at the file.
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
But it is not like this when I use a RHEL6 as a client to those servers, in this case, the ls only last for some seconds, nothing like the minutes that it takes from my fedora.
Which Fedora systems, exactly? The fix I described below is almost certainly in RHEL 6.
Fedora Core 14, 64 bit, 2.6.35.6-45
Right, you mentioned that in your OP. Sorry.
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
Dirty data is data that your application has written to the file but which hasn't been flushed to the server's disk. This data resides in the client's page cache, on its way to the server.
ok, understood. Then the sysctl change that you suggest, I've been checking both distributions, RHEL6 and FC14 and they share the same value... I assume by this that changing this value will not "help", am I right?
It should improve behavior somewhat in both cases, but the delay won't go away entirely. This was a workaround we gave EL5 customers before this bug was addressed. In the Fedora case I wouldn't expect a strongly deterministic improvement, but the average wait for "ls -l" should go down somewhat.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 6 lines · Chuck Lever
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
Can you help me on this? I'm I wrong in my supposition (is the patch really applied)? is it possible that my problem is somewhere else?
This sounds like typical behavior.
But it is not like this when I use a RHEL6 as a client to those servers, in this case, the ls only last for some seconds, nothing like the minutes that it takes from my fedora.
Which Fedora systems, exactly? The fix I described below is almost certainly in RHEL 6.
Fedora Core 14, 64 bit, 2.6.35.6-45
Right, you mentioned that in your OP. Sorry.
no problem
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
Dirty data is data that your application has written to the file but which hasn't been flushed to the server's disk. This data resides in the client's page cache, on its way to the server.
ok, understood. Then the sysctl change that you suggest, I've been checking both distributions, RHEL6 and FC14 and they share the same value... I assume by this that changing this value will not "help", am I right?
It should improve behavior somewhat in both cases, but the delay won't go away entirely. This was a workaround we gave EL5 customers before this bug was addressed. In the Fedora case I wouldn't expect a strongly deterministic improvement, but the average wait for "ls -l" should go down somewhat.
I saw that the value was 20, I don't know the impact of changing the number by units or tens... Should I test with 10 or this is too much? I assume that the behavior will change immediately right? j
Quoted 11 lines · Judith Flo Gaya
Quoted 7 lines · Chuck Lever
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
POSIX requires that the mtime and file size returned by stat(2) ('ls -l') reflect the most recent write(2). On NFS, the server sets both of these fields. If a client is caching dirty data, and an application does a stat(2), the client is forced to flush the dirty data so that the server can update mtime and file size appropriately. The client then does a GETATTR, and returns those values to the requesting application.
ok, sorry, I know this is a very stupid question but. what do you mean by dirty data?
Dirty data is data that your application has written to the file but which hasn't been flushed to the server's disk. This data resides in the client's page cache, on its way to the server.
ok, understood. Then the sysctl change that you suggest, I've been checking both distributions, RHEL6 and FC14 and they share the same value... I assume by this that changing this value will not "help", am I right?
It should improve behavior somewhat in both cases, but the delay won't go away entirely. This was a workaround we gave EL5 customers before this bug was addressed. In the Fedora case I wouldn't expect a strongly deterministic improvement, but the average wait for "ls -l" should go down somewhat.
I saw that the value was 20, I don't know the impact of changing the number by units or tens... Should I test with 10 or this is too much? I assume that the behavior will change immediately right?
I believe the dirty ratio is the percentage of physical memory that can be consumed by one file's dirty data before the VM starts flushing its pages asynchronously. Or it could be the amount of dirty data allowed across all files... one file or many doesn't make any difference if you are writing a single very large file.
If your client memory is large, a small number should work without problem. One percent of a 16GB client is still quite a bit of memory. The current setting means you can have 20% of said 16GB client, or 3.2GB, of dirty file data on that client before it will even think about flushing it. Along comes "ls -l" and you will have to wait for the client to flush 3.2GB before it can send the GETATTR.
I believe this setting does take effect immediately, but you will have to put the setting in /etc/sysctl.conf to make it last across a reboot.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hello Chuck,
Quoted 4 lines · Chuck Lever
Quoted 1 line · Judith Flo Gaya
Quoted 8 lines
Quoted 1 line
I saw that the value was 20, I don't know the impact of changing the number by units or tens... Should I test with 10 or this is too much? I assume that the behavior will change immediately right?
I believe the dirty ratio is the percentage of physical memory that can be consumed by one file's dirty data before the VM starts flushing its pages asynchronously. Or it could be the amount of dirty data allowed across all files... one file or many doesn't make any difference if you are writing a single very large file.
If your client memory is large, a small number should work without problem. One percent of a 16GB client is still quite a bit of memory. The current setting means you can have 20% of said 16GB client, or 3.2GB, of dirty file data on that client before it will even think about flushing it. Along comes "ls -l" and you will have to wait for the client to flush 3.2GB before it can send the GETATTR.
I believe this setting does take effect immediately, but you will have to put the setting in
/etc/sysctl.confto make it last across a reboot.
I made some tests with a value of 10 for the vm_dirty_ratio and indeed the ls-hang-time has decreased a lot, from 3min avg to 1.5min.
I was wondering what is the minimum number that it is safe to use? I'm sure that you have already dealt with the side-effects/collateral damages of this action, I don't want to fix a problem creating another one..
Regarding the modification of the inode.c file, what do you think that will be the next step? And how can I apply it to my system? Should I modify the file by myself and recompile the kernel to have the changed applied?
Thanks a lot,
j
Quoted 11 lines · Judith Flo Gaya
Hello Chuck,
Quoted 8 lines · Chuck Lever
Quoted 2 lines · Judith Flo Gaya
I saw that the value was 20, I don't know the impact of changing the number by units or tens... Should I test with 10 or this is too much? I assume that the behavior will change immediately right?
I believe the dirty ratio is the percentage of physical memory that can be consumed by one file's dirty data before the VM starts flushing its pages asynchronously. Or it could be the amount of dirty data allowed across all files... one file or many doesn't make any difference if you are writing a single very large file.
Quoted 7 lines
Quoted 4 lines
If your client memory is large, a small number should work without problem. One percent of a 16GB client is still quite a bit of memory. The current setting means you can have 20% of said 16GB client, or 3.2GB, of dirty file data on that client before it will even think about flushing it. Along comes "ls -l" and you will have to wait for the client to flush 3.2GB before it can send the GETATTR.
I believe this setting does take effect immediately, but you will have to put the setting in
/etc/sysctl.confto make it last across a reboot.I made some tests with a value of 10 for the
vm_dirty_ratioand indeed the ls-hang-time has decreased a lot, from 3min avg to 1.5min.
I was wondering what is the minimum number that it is safe to use? I'm sure that you have already dealt with the side-effects/collateral damages of this action, I don't want to fix a problem creating another one..
As I said before, you can set it to 1, and that will mean background flushing kicks in at 1% of your client's physical memory. I think that's probably safe nearly anywhere, but it may have deleterious effects on workload performance. You need to test various settings with your workload and your clients to see what is the best setting in your environment.
Quoted 1 line
Regarding the modification of the
inode.cfile, what do you think that will be the next step? And how can I apply it to my system? Should I modify the file by myself and recompile the kernel to have the changed applied?
I recommend that you file a bug against Fedora 14. See http://bugzilla.redhat.com/ .
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Yes, "may have deleterious effects on workload performance", is true. Especially the "may" part. Many applications simply open a file, write it out, and then close it. The aggressive caching really only helps when an application is working within an open file, where the working set fits within the cache, for an extended period. Otherwise, the client may as well start flushing as soon as it can generate full WRITE requests and as long as it is doing that, it may as well throttle the application to match the rate in which pages can be cleaned.
This will help with overall performance by not tying down memory to hold dirty pages which will not get touched again until they are clean. It can also greatly help the "ls -l" problem.
ps
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chuck Lever
Sent: Wednesday, March 16, 2011 9:25 AM
To: Judith Flo Gaya
Cc: [email protected]
Subject: Re: problem with nfs latency during high IO
Quoted 11 lines · Judith Flo Gaya
Hello Chuck,
Quoted 8 lines · Chuck Lever
Quoted 2 lines · Judith Flo Gaya
I saw that the value was 20, I don't know the impact of changing the number by units or tens... Should I test with 10 or this is too much? I assume that the behavior will change immediately right?
I believe the dirty ratio is the percentage of physical memory that can be consumed by one file's dirty data before the VM starts flushing its pages asynchronously. Or it could be the amount of dirty data allowed across all files... one file or many doesn't make any difference if you are writing a single very large file.
Quoted 7 lines
Quoted 4 lines
If your client memory is large, a small number should work without problem. One percent of a 16GB client is still quite a bit of memory. The current setting means you can have 20% of said 16GB client, or 3.2GB, of dirty file data on that client before it will even think about flushing it. Along comes "ls -l" and you will have to wait for the client to flush 3.2GB before it can send the GETATTR.
I believe this setting does take effect immediately, but you will have to put the setting in
/etc/sysctl.confto make it last across a reboot.I made some tests with a value of 10 for the
vm_dirty_ratioand indeed the ls-hang-time has decreased a lot, from 3min avg to 1.5min.
I was wondering what is the minimum number that it is safe to use? I'm sure that you have already dealt with the side-effects/collateral damages of this action, I don't want to fix a problem creating another one..
As I said before, you can set it to 1, and that will mean background flushing kicks in at 1% of your client's physical memory. I think that's probably safe nearly anywhere, but it may have deleterious effects on workload performance. You need to test various settings with your workload and your clients to see what is the best setting in your environment.
Quoted 1 line
Regarding the modification of the
inode.cfile, what do you think that will be the next step? And how can I apply it to my system? Should I modify the file by myself and recompile the kernel to have the changed applied?
I recommend that you file a bug against Fedora 14. See http://bugzilla.redhat.com/ .
Signature · 10 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Chuck Lever
I was wondering what is the minimum number that it is safe to use? I'm
sure that you have already dealt with the side-effects/collateral damages of this action, I don't want to fix a problem creating another one..
As I said before, you can set it to 1, and that will mean background flushing kicks in at 1% of your client's physical memory.
I think 5 is the minimum for dirty_ratio, although this doesn't seem to be documented anywhere. If you want to set it lower, you have to use dirty_bytes instead of dirty_ratio. See the commit message here:
http://lkml.org/lkml/2008/11/23/160
Documentation is in Documentation/sysctl/vm.txt.
Jim Rees wrote:
I think 5 is the minimum for dirty_ratio, although this doesn't seem to be documented anywhere. If you want to set it lower, you have to use dirty_bytes instead of dirty_ratio.
Never mind, I just looked at the code and the 5% limit seems to have been removed later. Last time I looked at this code was 3 years ago.
THanks a lot to both for your suggestions and help. I already filed a bug [1] to bugzilla.
j
[1] https://bugzilla.redhat.com/show_bug.cgi?id=688232
Quoted 8 lines · Jim Rees
Jim Rees wrote:
I think 5 is the minimum for
dirty_ratio, although this doesn't seem to be documented anywhere. If you want to set it lower, you have to usedirty_bytesinstead ofdirty_ratio.Never mind, I just looked at the code and the 5% limit seems to have been removed later. Last time I looked at this code was 3 years ago.
Signature · 11 lines
Judith Flo Gaya Systems Administrator IMPPC e-mail: [email protected] Tel (+34) 93 554-3079 Fax (+34) 93 465-1472 Institut de Medicina Predictiva i Personalitzada del Càncer Crta Can Ruti, Camí de les Escoles s/n 08916 Badalona, Barcelona, Spain http://www.imppc.org
Quoted 6 lines · Judith Flo Gaya
I made some tests with a value of 10 for the
vm_dirty_ratioand indeed the ls-hang-time has decreased a lot, from 3min avg to 1.5min. I was wondering what is the minimum number that it is safe to use? I'm sure that you have already dealt with the side-effects/collateral damages of this action, I don't want to fix a problem creating another one..
For a while, we were running with this on production NFS clients:
vm/dirty_background_bytes = 1048576
vm/dirty_bytes = 2097152
which is totally crazy, but generally seemed to work for the most part, and penalized the process creating the pages instead of totally hosing everything else when somebody was just writing back a huge file. Without it, we were seeing a single "dd" able to cause the load to hit 200, simply because 200 other processes got stuck in D waiting for RPC slots due to the slowdown. With those settings, load would stay around 3-4 and latency was much better.
I think I removed it when trying to figure out the other flush issues, but it seems the same problem still exists.
Simon-