kernel 2.6.13 buffer strangeness
Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.
regards, Anthony
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.
And I think that you mean MB(yte) not Mb(it).
Roger
Quoted 33 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangenessThanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering
/proc/sys/vm/dirty_ratio, it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.regards, Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd[email protected]
Phone: (02) 62595404 or 0419409836-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Roger Heflin
I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.
yes, on 2.6 there are two tunables which are important here. dirty_background_ratio is the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes active. dirty_ratio changes when the write-throttling code kicks in, which is what Anthony is seeing. The purpose of the write throttling code is to limit the dirtying process to disk bandwidth, so that is a Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100, but you could trigger an OOM if you let it get too high, so maybe try setting it at 85 or so. This should effectively disable the write throttling and give you the bandwidth you want.
NATE
Quoted 44 lines
And I think that you mean MB(yte) not Mb(it).
Roger
Quoted 26 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangenessThanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering
/proc/sys/vm/dirty_ratio, it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.regards, Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd[email protected]
Phone: (02) 62595404 or 0419409836Quoted 7 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi Nate and Roger,
First off, thanks fo taking some time to answer my call for help :-)
I've already spent some time fiddling with dirty_ratio before I posted my question - yes, I can make different things happen by increasing or decreasing it, but overall I still have the same problem - my video streaming runs out of steam after at most 50 seconds instead of the 3 minutes that I think it should take.
Setting dirty_ratio and dirty_background_ratio to 15/10 respectively makes things worse, I hit the wall after only about 25 seconds.
Setting dirty_ratio and dirty_background_ratio to 85/80 makes things worse - disk writes start after about 15 seconds and I hit the wall after about 35 seconds.
Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.
So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.
Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.
So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.
Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.
Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??
regards, Anthony
Quoted 61 lines · Nate Diller
Quoted 8 lines · Roger Heflin
I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.
yes, on 2.6 there are two tunables which are important here.
dirty_background_ratiois the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes
active.dirty_ratiochanges when the write-throttling code kicks in,
which is what Anthony is seeing. The purpose of the write throttling
code is to limit the dirtying process to disk bandwidth, so that is a
Feature. Anthony, try *increasing*dirty_ratio, you can go up to 100,
but you could trigger an OOM if you let it get too high, so maybe try
setting it at 85 or so. This should effectively disable the write
throttling and give you the bandwidth you want.NATE
Quoted 35 lines
And I think that you mean MB(yte) not Mb(it).
Roger
Quoted 16 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangenessThanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering
/proc/sys/vm/dirty_ratio, it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.regards, Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty LtdQuoted 7 lines
[email protected]
Phone: (02) 62595404 or 0419409836-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Quoted 4 lines · Anthony Wesley
Hi Nate and Roger,
First off, thanks fo taking some time to answer my call for help :-)
Quoted 9 lines
I've already spent some time fiddling with
dirty_ratiobefore I posted my question - yes, I can make different things happen by increasing or decreasing it, but overall I still have the same problem - my video streaming runs out of steam after at most 50 seconds instead of the 3 minutes that I think it should take.Setting
dirty_ratioanddirty_background_ratioto 15/10 respectively makes things worse, I hit the wall after only about 25 seconds.
not surprising
Quoted 4 lines
Setting
dirty_ratioanddirty_background_ratioto 85/80 makes things worse - disk writes start after about 15 seconds and I hit the wall after about 35 seconds.
so dirty_background_ratio is working, that's why it took a long time to start writing anything out
Quoted 3 lines
Setting
dirty_ratioanddirty_background_ratioto 90/10 puts me back at around 50 seconds, i.e. where I started.
this setting should do the trick, so there's something going on here that isn't expected.
Quoted 10 lines
So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.
Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.
Quoted 3 lines
So, I've tried all the combinations on
dirty_ratioanddirty_background_ratioand they *do not help*.
dirty_ratio is the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bug
Quoted 3 lines
Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.
how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?
just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk
Quoted 4 lines
Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??
this should be possible, although it could be considered a bit risky WRT OOM.
NATE
Quoted 87 lines
regards, Anthony
Quoted 22 lines · Nate Diller
Quoted 8 lines · Roger Heflin
I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.
yes, on 2.6 there are two tunables which are important here.
dirty_background_ratiois the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes active.dirty_ratiochanges when the write-throttling code kicks in, which is what Anthony is seeing. The purpose of the write throttling code is to limit the dirtying process to disk bandwidth, so that is a Feature. Anthony, try *increasing*dirty_ratio, you can go up to 100, but you could trigger an OOM if you let it get too high, so maybe try setting it at 85 or so. This should effectively disable the write throttling and give you the bandwidth you want.Quoted 44 lines
NATE
Quoted 41 lines
And I think that you mean MB(yte) not Mb(it).
Roger
Quoted 26 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangenessThanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering
/proc/sys/vm/dirty_ratio, it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.regards, Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd[email protected]
Phone: (02) 62595404 or 0419409836Quoted 7 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"in
Quoted 6 lines
Quoted 4 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Based on a better understanding of dirty_ratio and dirty_background ratio (thanks Nate) I just tried the following test:
dirty_ratio set to 95dirty_background_ratio set to 1
Quoted 1 line
From Nate's description of these parameters, this should mean that the disk writes
start almost immediately, and the kernel will allow 95% of RAM to become dirty before applying the throttle.
Ok, so with 25Mbytes/s coming in, and 17Mbytes/sec going out to disk, the dirty pages should be growing at 7Mbytes/sec. With these parameters set as above I should see about 3 minutes of full speed video before the throttle is applied since I have about 1.3Gb of RAM free for buffering..
*But* when I try this experiment I hit the throttle after only 65 seconds - an improvement to be sure, but still a long way short of the 180 seconds that it ought to take.
Part of the test works as expected - the disk writes begin almost immediately due to the low value for dirty_background_ratio, but the rest is a mystery.
It really looks as if the pages aren't being marked as clean fast enough after they're written.
How else can it take only 70 seconds to reach 95% dirty when I have 1.3Gb of available RAM and data coming in at 25MBytes/sec and out at 17MBytes/sec? It doesn't make any sense...
regards, Anthony
Quoted 11 lines · Nate Diller
yes, on 2.6 there are two tunables which are important here.
dirty_background_ratiois the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes
active.dirty_ratiochanges when the write-throttling code kicks in,
which is what Anthony is seeing. The purpose of the write throttling
code is to limit the dirtying process to disk bandwidth, so that is a
Feature. Anthony, try *increasing*dirty_ratio, you can go up to 100,
but you could trigger an OOM if you let it get too high, so maybe try
setting it at 85 or so. This should effectively disable the write
throttling and give you the bandwidth you want.NATE
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Okay, just tested a couple of things, here's what I see...
I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop hard drive, only 5400rpm so this is not surprising.
I did this test with my video capture app running, but just displaying data and not writing,
I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's a 4200rpm drive so again not surprising numbers.
The video data is written as individual frames, so the efficiency is a bit below the raw throughput, but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would write its data.
I'd bet a large sum of money that these hard disk figures are correct to within a few percent.
Also, when I am actually capturing I have timed by stopwatch how long the disk activity light is on, and reached about the same conclusion.
Working the problem from the other direction, the only way to explain the early throttling that I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.
regards, Anthony
Quoted 47 lines · Nate Diller
Quoted 3 lines
Setting
dirty_ratioanddirty_background_ratioto 90/10 puts me back at around 50 seconds, i.e. where I started.this setting should do the trick, so there's something going on here that isn't expected.
Quoted 10 lines
So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.
Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.
Quoted 3 lines
So, I've tried all the combinations on
dirty_ratioanddirty_background_ratioand they *do not help*.
dirty_ratiois the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bugQuoted 3 lines
Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.
how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?
just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk
Quoted 4 lines
Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??
this should be possible, although it could be considered a bit risky WRT OOM.
NATE
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Quoted 35 lines · Anthony Wesley
Okay, just tested a couple of things, here's what I see...
I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop
hard drive, only 5400rpm so this is not surprising.I did this test with my video capture app running, but just displaying data and not writing,
I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's
a 4200rpm drive so again not surprising numbers.The video data is written as individual frames, so the efficiency is a bit below the raw throughput,
but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would
write its data.I'd bet a large sum of money that these hard disk figures are correct to within a few percent.
Also, when I am actually capturing I have timed by stopwatch how long the disk activity light
is on, and reached about the same conclusion.Working the problem from the other direction, the only way to explain the early throttling that
I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if
the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds
of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.
Quoted 2 lines
regards, Anthony
i need a vmstat trace ('vmstat 1 > logfile', then run the test, then ^c) before i can comment further. also, your filesystem and scheduler would be interesting to know. you can find out the scheduler with 'cat /sys/block/[drive]/queue/scheduler'.
NATE
Quoted 67 lines
Quoted 9 lines · Nate Diller
Quoted 3 lines
Setting
dirty_ratioanddirty_background_ratioto 90/10 puts me back at around 50 seconds, i.e. where I started.this setting should do the trick, so there's something going on here that isn't expected.
Quoted 1 line
So as far as I can see there is *no way* to get 3 minutes worth of
buffering
Quoted 3 lines
Quoted 3 lines
by adjusting these parameters.
Just to remind everyone - I have video data coming in at 25MBytes/sec and
I
Quoted 1 line
Quoted 1 line
am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I
want
Quoted 2 lines
Quoted 2 lines
my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I
have
Quoted 1 line
Quoted 1 line
1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes
to
Quoted 14 lines
Quoted 5 lines
consume at 7MBytes/sec.
So, I've tried all the combinations on
dirty_ratioanddirty_background_ratioand they *do not help*.
dirty_ratiois the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bugQuoted 3 lines
Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.
Quoted 10 lines
how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?
just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk
Quoted 1 line
Or is this simply not possible with the current kernel I/O setup? Do I
have
Quoted 5 lines
Quoted 3 lines
to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??
this should be possible, although it could be considered a bit risky WRT
OOM.
Quoted 2 lines
NATE
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Nate, this test was done with the cfq scheduler, but I've tried them all, and they didn't seem to make any real difference. Let me know if you want me to re-run this with one of the other schedulers. The default scheduler is anticipatory, and that's where the problem first showed up.
Attached is the output of "vmstat -S K 1". I started the capture a couple of seconds after starting this trace and stopped it after 2 minutes.
If I'm interpreting it correctly, this shows a disk throughput averaging about 15-16MBytes/sec.
I saw the video write speed sustain 25Mbytes/sec for about 75 seconds and then it dropped.
dirty_background_ratio is set to 1dirty_ratio is set to 95
Assuming that these numbers are ok, that indicates a deficit of about 10MBytes/sec between my video in and disk out, so for 1.3Gb of ram as buffer I should see about 130 seconds at full speed before it drops. Instead I see this happen after about 75 seconds.
Thanks again,
Anthony
Quoted 41 lines · Nate Diller
Quoted 34 lines · Anthony Wesley
Okay, just tested a couple of things, here's what I see...
I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop hard drive, only 5400rpm so this is not surprising.
I did this test with my video capture app running, but just displaying data and not writing,
I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's
a 4200rpm drive so again not surprising numbers.The video data is written as individual frames, so the efficiency is a bit below the raw throughput,
but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would
write its data.I'd bet a large sum of money that these hard disk figures are correct to within a few percent.
Also, when I am actually capturing I have timed by stopwatch how long the disk activity light
is on, and reached about the same conclusion.Working the problem from the other direction, the only way to explain the early throttling that
I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if
the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds
of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.Quoted 4 lines
regards, Anthony
Quoted 121 lines
i need a vmstat trace ('vmstat 1 > logfile', then run the test, then ^c) before i can comment further. also, your filesystem and scheduler would be interesting to know. you can find out the scheduler with 'cat /sys/block/[drive]/queue/scheduler'.
NATE
Quoted 109 lines
Quoted 5 lines · Nate Diller
Quoted 5 lines
Setting
dirty_ratioanddirty_background_ratioto 90/10 puts me back at around 50 seconds, i.e. where I started.Quoted 9 lines
this setting should do the trick, so there's something going on here that isn't expected.
Quoted 3 lines
So as far as I can see there is *no way* to get 3 minutes worth of
buffering
Quoted 5 lines
Quoted 5 lines
by adjusting these parameters.
Just to remind everyone - I have video data coming in at 25MBytes/sec and
I
Quoted 3 lines
Quoted 3 lines
am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I
want
Quoted 4 lines
Quoted 4 lines
my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I
have
Quoted 3 lines
Quoted 3 lines
1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes
to
Quoted 18 lines
Quoted 7 lines
consume at 7MBytes/sec.
So, I've tried all the combinations on
dirty_ratioanddirty_background_ratioand they *do not help*.
dirty_ratiois the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bugQuoted 5 lines
Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.
Quoted 14 lines
how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?
just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk
Quoted 3 lines
Or is this simply not possible with the current kernel I/O setup? Do I
have
Quoted 8 lines
Quoted 5 lines
to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??
this should be possible, although it could be considered a bit risky WRT
OOM.
Quoted 3 lines
NATE
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 4 0 144 1314396 29440 67444 0 0 16 407 290 17 30 30 38 2 0 0 144 1314396 29440 67436 0 0 0 0 405 1062 26 4 70 0 0 0 144 1314388 29440 67436 0 0 0 0 721 1397 27 4 69 0 0 0 144 1314388 29440 67436 0 0 0 0 732 1554 30 3 67 0 0 0 144 1300624 29548 80220 0 0 24 0 426 1207 26 13 40 21 0 0 144 1275444 29744 105148 0 0 8 20752 933 2262 31 21 49 0 3 2 144 1250396 29912 130076 0 0 4 15286 1004 2357 31 22 0 47 0 3 144 1225224 30076 155004 0 0 0 14008 1030 2439 28 22 0 49 1 3 144 1200788 30244 179324 0 0 8 13772 1049 2526 29 21 0 50 0 3 144 1175616 30412 204256 0 0 4 15964 1102 2696 20 18 0 62 0 5 144 1150552 30584 229184 0 0 0 15980 1100 2724 21 17 0 62 2 1 144 1125836 30756 253504 0 0 12 14672 1106 2715 20 19 0 62 0 7 144 1100788 30920 278432 0 0 0 15380 1079 2665 19 18 0 63 2 6 144 1075616 31092 303360 0 0 8 15440 1112 2731 21 18 0 61 0 8 144 1049948 31264 328900 0 0 4 14832 1063 2685 18 22 0 60 2 8 144 1024776 31436 353828 0 0 0 15620 1117 2815 19 17 0 64 0 2 144 1001588 31600 376932 0 0 12 15828 1093 2677 18 20 0 62 2 6 144 976432 31764 401860 0 0 0 14724 1085 2604 21 19 0 60 0 5 144 950764 31932 427396 0 0 0 15844 1093 2734 21 18 0 61 1 4 144 924964 32112 452936 0 0 12 15028 1044 2583 22 18 0 61 0 4 144 899792 32292 477864 0 0 8 14788 1071 2670 18 18 0 64 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 2 144 880200 32432 497320 0 0 12 15228 1103 2676 18 18 0 64 0 5 144 855152 32596 522248 0 0 0 14604 1053 2598 19 20 0 61 0 6 144 829980 32760 547176 0 0 0 16004 1124 2741 21 17 0 63 0 6 144 804188 32932 572716 0 0 4 15896 1081 2704 19 18 0 63 1 4 144 779116 33112 597644 0 0 8 15020 1094 2722 19 19 0 62 0 11 144 755672 33272 620840 0 0 4 15876 1076 2750 17 20 0 63 1 11 144 733104 33416 643244 0 0 0 16220 1110 2795 19 16 0 65 0 2 144 708056 33580 668172 0 0 0 15836 1074 2668 20 18 0 63 0 8 144 682884 33744 693104 0 0 0 15952 1115 2758 20 16 0 64 1 4 144 657216 33920 718640 0 0 0 15908 1115 2765 18 18 0 64 0 6 144 632664 34088 742960 0 0 8 14928 1080 2663 21 17 0 62 0 3 144 609972 34252 765456 0 0 16 14888 1055 2606 20 19 0 61 1 2 144 584320 34432 790992 0 0 12 15044 1094 2709 18 20 0 62 0 1 144 559768 34600 815316 0 0 8 14948 1069 2634 20 19 0 61 0 6 144 534596 34772 840244 0 0 0 16188 1092 2645 20 20 0 60 1 5 144 508928 34944 865780 0 0 4 14832 1063 2625 21 20 0 60 0 2 144 489972 35080 884628 0 0 12 13700 1057 2515 21 18 0 61 1 3 144 464800 35256 909556 0 0 12 17008 1101 2728 23 15 0 62 1 1 144 439008 35432 935096 0 0 8 13692 1109 2708 22 18 0 60 0 3 144 413960 35604 960024 0 0 0 17072 1096 2676 19 20 0 61 1 5 144 388804 35776 984952 0 0 8 14484 1082 2640 22 20 0 59 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 2 144 366112 35936 1007448 0 0 12 15324 1078 2626 20 21 0 59 2 2 144 341064 36108 1032376 0 0 8 14544 1060 2551 22 18 0 60 1 3 144 315272 36276 1057912 0 0 0 15836 1057 2639 22 20 0 59 0 5 144 290100 36448 1082844 0 0 0 15420 1130 2764 21 19 0 60 1 5 144 264060 36616 1108384 0 0 0 14836 1078 2648 23 18 0 59 0 5 144 238640 36780 1133312 0 0 0 16124 1109 2725 20 20 0 60 2 4 144 215544 36932 1155808 0 0 4 14768 1058 2613 21 18 0 61 0 9 144 189504 37104 1181344 0 0 4 14064 1125 2823 21 21 0 58 1 6 144 164084 37276 1206276 0 0 0 16040 1071 2728 19 20 0 61 1 9 144 138664 37440 1231204 0 0 0 16020 1136 2822 20 20 0 60 1 1 144 113120 37608 1256132 0 0 4 14604 1070 2734 18 20 0 62 2 6 144 89560 37764 1279236 0 0 4 14896 1081 2612 17 22 0 61 0 9 144 64140 37928 1304164 0 0 0 17184 1092 2818 18 21 0 61 0 9 144 38572 38100 1329096 0 0 0 15784 1126 2833 22 19 0 59 2 3 144 32628 27604 1345272 0 0 0 14780 1101 2675 21 23 0 56 1 9 144 32628 16548 1356284 0 0 0 16024 1100 2740 21 22 0 57 3 9 144 32628 16712 1356028 0 0 4 14852 1069 2729 19 19 0 62 0 4 144 32628 16880 1355844 0 0 8 15044 1082 2692 19 21 0 60 1 9 144 32132 17052 1356200 0 0 0 16100 1090 2776 21 20 0 59 2 6 144 32504 17216 1355784 0 0 0 14712 1097 2719 20 22 0 58 0 9 144 32628 17384 1355368 0 0 4 14908 1037 2690 22 20 0 58 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 5 144 32380 17536 1355528 0 0 12 13812 1060 2593 24 17 0 59 0 9 144 32752 17704 1354952 0 0 0 14932 1063 2654 22 23 0 55 0 9 144 32256 17880 1355244 0 0 0 7228 1113 2881 22 21 0 57 0 2 144 32008 18044 1355388 0 0 8 14892 1090 2761 23 22 0 56 1 3 144 32008 18220 1355052 0 0 8 13996 1050 2585 20 22 0 58 1 9 144 32504 18360 1354524 0 0 8 13252 1021 2530 19 19 0 62 1 8 144 32876 18516 1353884 0 0 0 15956 1120 2771 22 18 0 60 1 6 144 32804 18700 1353760 0 0 8 14120 1079 2738 23 22 0 56 0 8 144 32628 18868 1353748 0 0 4 12808 1221 3006 21 22 0 57 1 8 144 33052 19024 1353080 0 0 0 6320 989 2631 21 21 0 58 0 8 144 32756 19192 1353224 0 0 0 6308 1099 2833 21 22 0 57 0 2 144 32628 19300 1353296 0 0 12 10540 1019 2564 20 18 0 62 2 8 144 31992 19428 1353644 0 0 4 20276 973 2448 20 19 0 61 0 8 144 32256 19520 1353376 0 0 4 9816 1103 2694 19 20 0 62 1 4 144 32028 19600 1353384 0 0 4 18116 1114 2643 19 16 0 65 1 5 144 33000 19740 1352244 0 0 4 12424 1055 2593 19 21 0 60 0 8 144 33076 19848 1352068 0 0 4 22344 1111 2682 18 17 0 65 2 3 144 32480 19964 1352520 0 0 12 9428 1078 2709 18 17 0 65 1 8 144 32224 20080 1352584 0 0 4 17816 1071 2622 17 21 0 62 0 11 144 32224 20184 1352604 0 0 4 14052 1054 2643 19 17 0 64 1 13 144 32052 20292 1352540 0 0 4 15940 1110 2752 21 17 0 63 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 13 144 32596 20384 1351948 0 0 4 14792 1041 2631 21 14 0 65 2 14 144 32720 20456 1351652 0 0 4 18476 1077 2676 19 14 0 67 0 5 144 32720 20604 1351568 0 0 0 13072 1078 2692 21 18 0 61 0 10 144 32720 20724 1351336 0 0 4 19720 1106 2690 20 18 0 62 1 13 144 31976 20816 1352016 0 0 4 11964 1065 2639 19 15 0 66 0 14 144 32596 20876 1351476 0 0 8 19012 1092 2702 19 17 0 65 1 14 144 31976 21012 1351784 0 0 8 13204 1084 2805 20 19 0 61 0 2 144 31976 21100 1351776 0 0 0 14896 1078 2706 20 16 0 64 0 9 144 32476 21240 1351048 0 0 4 18760 1069 2603 20 23 0 58 1 13 144 32100 21344 1351392 0 0 0 14720 1126 2774 20 17 0 63 0 14 144 32596 21460 1350860 0 0 4 18992 1095 2735 22 18 0 60 0 14 144 33096 21576 1350072 0 0 0 14660 1118 2860 22 17 0 61 1 13 144 31976 21656 1351056 0 0 4 13764 1074 2674 20 18 0 63 0 4 144 32724 21788 1350176 0 0 0 19236 1089 2710 20 20 0 60 1 9 144 32720 21888 1350128 0 0 4 12440 1079 2604 20 20 0 61 0 14 144 32348 21996 1350320 0 0 8 24964 1111 2753 21 15 0 64 0 13 144 32704 22112 1349948 0 0 4 7960 1088 2791 17 22 0 61 2 14 144 32208 22196 1350360 0 0 0 22596 1136 2790 20 21 0 60 0 2 144 32456 22328 1349764 0 0 4 9816 1049 2681 21 19 0 60 0 8 144 32084 22448 1350124 0 0 0 17024 1105 2668 19 21 0 60 2 13 144 32084 22544 1350000 0 0 0 19064 1102 2740 18 20 0 62 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 4 144 32828 22668 1349180 0 0 8 18948 1106 2710 18 17 0 65 1 7 144 31960 22768 1349892 0 0 8 10664 1068 2580 18 21 0 62 1 12 144 32084 22892 1349488 0 0 0 18428 1118 2753 21 18 0 61 0 13 144 31960 22996 1349656 0 0 4 14540 1082 2713 19 17 0 64 0 14 144 32084 23088 1349364 0 0 4 23028 1089 2751 17 17 0 66 1 13 144 31960 23228 1349220 0 0 0 8720 1103 2785 19 19 0 62 0 13 144 32580 23320 1348772 0 0 4 14816 1053 2626 20 17 0 64 0 13 144 32208 23404 1348964 0 0 4 15972 1068 2628 18 16 0 66 1 14 144 32580 23380 1348432 0 0 0 25216 1106 2808 20 19 0 61 0 13 144 32704 23144 1348680 0 0 4 7140 1099 2742 17 17 0 66 0 13 144 32580 22776 1349172 0 0 4 22132 1128 2822 24 15 0 62 2 14 144 32332 22192 1349820 0 0 0 14152 1062 2691 18 21 0 61 0 13 144 32208 21628 1350684 0 0 0 15448 1121 2750 18 18 0 64 1 5 144 31960 20996 1351536 0 0 4 18192 1038 2541 21 21 0 58 0 9 144 32208 21024 1351088 0 0 0 11980 1119 2689 19 19 0 62 1 13 144 32828 21100 1350576 0 0 0 19696 1095 2783 21 14 0 65 0 13 144 32084 21160 1351156 0 0 4 18324 1097 2769 22 16 0 62 1 13 144 32832 21276 1350352 0 0 4 10688 1076 2709 20 18 0 62 0 13 144 32952 21388 1350604 0 0 4 20564 1124 2843 22 18 0 60 2 1 144 32952 21392 1350604 0 0 4 14460 1146 2729 17 15 0 68 0 1 144 32952 21404 1350604 0 0 4 16548 1402 3150 19 12 0 69 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 1 144 32952 21408 1350604 0 0 4 18056 1167 2657 16 14 0 70 0 1 144 32952 21408 1350604 0 0 0 18548 1189 2675 17 17 0 67 0 1 144 32952 21412 1350604 0 0 4 14948 1067 2490 17 12 0 71
Quoted 1 line · Nate Diller
just found the culprit. guess i should have read the code the first time.
get_dirty_limits()indrivers/block/page_writeback.chas a hard-coded upper limit todirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio>unmapped_ratio/ 2)dirty_ratio=unmapped_ratio/ 2;' check, and you can have all the OOM goodness you want.
Excellent. OOM here I come.
Quoted 1 line
i really recommend you focus on getting better disk bandwidth, you stand to gain a lot more from that approach. i presume you're on ext3; perhaps you should try reiser4 or xfs, they are more likely to meet your disk bandwidth requirements.
Yep, pursuing this as well, also looking to add more RAM to the machine. At this stage I was just trying to understand the numbers that I was seeing, so I could work out the best way to proceed.
I'm using ext2 at the moment, on the assumption that the journal would cost me a bit of performance so I left it out :-)
I'll certainly try the other filesystems as you suggest.
Thanks again, Anthony
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Quoted 2 lines · Anthony Wesley
How else can it take only 70 seconds to reach 95% dirty when I have 1.3Gb of available RAM and data coming in at 25MBytes/sec and out at 17MBytes/sec? It doesn't make any sense...
What architecture? x86?
If so then bear in mind that your memory is split into 500MB highmem and 800MB lowmem. The kernel might be starting I/O due to the highmem zone being full of dirty pages. That'd be wrong of it if so - it's supposed to just fall back to lowmem for the page allocations, but that code has changed quite a bit in the two years since I got all that working...
You need to run `watch -n1 cat /proc/meminfo' while doing these tests...
Re: kernel 2.6.13 buffer strangeness - FIXED
In reply to a message not in the archiveQuoted 1 line · Nate Diller
just found the culprit. guess i should have read the code the first time.
get_dirty_limits()indrivers/block/page_writeback.chas a hard-coded upper limit todirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio>unmapped_ratio/ 2)dirty_ratio=unmapped_ratio/ 2;' check, and you can have all the OOM goodness you want.
Well well well.
Thank you Nate!
I changed that bit of code to:
if (dirty_ratio > unmapped_ratio - 10) dirty_ratio = unmapped_ratio - 10;
and added a couple of sanity checks so that it couldn't get below 5 or above 95.
Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.
Guess what? Now my video streams for 120 seconds before being throttled. This is *much better*. It's now quite feasible to reach 30 seconds by finding a faster disk and/or adding another 512M or RAM.
I guess the hard-coded limit is in there for a reason, but I sure wish it was adjustable without this sort of hacking.
regards, Anthony
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Re: kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison
In reply to a message not in the archiveQuoted 3 lines · Nate Diller
i really recommend you focus on getting better disk bandwidth, you stand to gain a lot more from that approach. i presume you're on ext3; perhaps you should try reiser4 or xfs, they are more likely to meet your disk bandwidth requirements.
NATE
While I have already solved the issue that was troubling me, I also spent some time comparing different filesystems as reccommended by Nate, with interesting results.
My method was simple - make a filesystem and the set it as the target for my video capture. With video coming in at 25MBytes/sec and going out to disk at about 15-20MBytes/sec it is an interesting test of the vm and filesystem.
I compared ext2,ext3,xfs,vfat,reiser and reiser4.
The hands-down winner was ext2. All the others showed problems of either lower disk throughput or dropped frames during video capture.
Only ext2 went the full distance - no dropped frames until we run out of RAM, and good disk throughput.
xfs,reiser and reiser4 had slightly higher disk write speed, but showed performance problems that caused lots of dropped frames so they must be ruled out at this stage.
I know that xfs and reiser4 are supposed to be faster for some things, but it seems to me that they are not the best choice when you are predominantly writing lots and lots of 600k files :-)
regards, Anthony
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836
Re: kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison
Quoted 5 lines · Anthony Wesley
I compared ext2,ext3,xfs,vfat,reiser and reiser4.
The hands-down winner was ext2. All the others showed problems of either lower disk throughput or dropped frames during video capture.
ext2 is a good filesystem. For that sort of application all the journaling gunk can really get in the way.
You should have tested ext3 with data=writeback.
Re: kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison
Quoted 4 lines · Andrew Morton
Quoted 2 lines · Anthony Wesley
I compared ext2,ext3,xfs,vfat,reiser and reiser4.
Quoted 9 lines
Quoted 2 lines
The hands-down winner was ext2. All the others showed problems of either lower disk throughput or dropped frames during video capture.
ext2 is a good filesystem. For that sort of application all the journaling gunk can really get in the way.
You should have tested ext3 with data=writeback.
Ask and ye shall receive...
I created an ext3 fs, mounted it with data=writeback and gave it a quick spin.
The result? Lots of pauses and dropped frames during capture. This is during the part of the process where I have gobs of free RAM that's being used for buffering so dropping frames here is a cardinal sin.
Dunno why it's happening, but I saw it also with xfs and reiser4. ext2 on the other hand chugs along happily, no pauses, no dropped frames until we run out of free RAM (takes about 2 minutes now after the simple kernel change).
I can understand dropped frames after we run out of ram, but not before.
regards, Anthony
Signature · 7 lines
Anthony Wesley Director and IT/Network Consultant Smart Networks Pty Ltd Acquerra Pty Ltd [email protected] Phone: (02) 62595404 or 0419409836