Skip to contentSkip to conversation

kernel 2.6.13 buffer strangeness

DISCUSSION Anthony Wesley 4 participants 15 messages Sep 9, 2005 – Sep 10, 2005 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.

regards, Anthony

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#1 lore raw

I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.

And I think that you mean MB(yte) not Mb(it).

Roger

Quoted 33 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangeness

Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.

regards, Anthony

--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 9 lines · Roger Heflin

I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.

yes, on 2.6 there are two tunables which are important here. dirty_background_ratio is the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes active. dirty_ratio changes when the write-throttling code kicks in, which is what Anthony is seeing. The purpose of the write throttling code is to limit the dirtying process to disk bandwidth, so that is a Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100, but you could trigger an OOM if you let it get too high, so maybe try setting it at 85 or so. This should effectively disable the write throttling and give you the bandwidth you want.

NATE

Quoted 44 lines

And I think that you mean MB(yte) not Mb(it).

Roger

Quoted 26 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangeness

Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.

regards, Anthony

--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836

Quoted 7 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw

Hi Nate and Roger,

First off, thanks fo taking some time to answer my call for help :-)

I've already spent some time fiddling with dirty_ratio before I posted my question - yes, I can make different things happen by increasing or decreasing it, but overall I still have the same problem - my video streaming runs out of steam after at most 50 seconds instead of the 3 minutes that I think it should take.

Setting dirty_ratio and dirty_background_ratio to 15/10 respectively makes things worse, I hit the wall after only about 25 seconds.

Setting dirty_ratio and dirty_background_ratio to 85/80 makes things worse - disk writes start after about 15 seconds and I hit the wall after about 35 seconds.

Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.

So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.

Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.

So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.

Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.

Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??

regards, Anthony

Quoted 61 lines · Nate Diller
Quoted 8 lines · Roger Heflin

I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.

yes, on 2.6 there are two tunables which are important here. dirty_background_ratio is the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes
active. dirty_ratio changes when the write-throttling code kicks in,
which is what Anthony is seeing. The purpose of the write throttling
code is to limit the dirtying process to disk bandwidth, so that is a
Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100,
but you could trigger an OOM if you let it get too high, so maybe try
setting it at 85 or so. This should effectively disable the write
throttling and give you the bandwidth you want.

NATE

Quoted 35 lines

And I think that you mean MB(yte) not Mb(it).

Roger

Quoted 16 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangeness

Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.

regards, Anthony

--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

Quoted 7 lines

[email protected]
Phone: (02) 62595404 or 0419409836

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#4 lore raw
Quoted 4 lines · Anthony Wesley

Hi Nate and Roger,

First off, thanks fo taking some time to answer my call for help :-)

Quoted 9 lines

I've already spent some time fiddling with dirty_ratio before I posted my question - yes, I can make different things happen by increasing or decreasing it, but overall I still have the same problem - my video streaming runs out of steam after at most 50 seconds instead of the 3 minutes that I think it should take.

Setting dirty_ratio and dirty_background_ratio to 15/10 respectively makes things worse, I hit the wall after only about 25 seconds.

not surprising

Quoted 4 lines

Setting dirty_ratio and dirty_background_ratio to 85/80 makes things worse - disk writes start after about 15 seconds and I hit the wall after about 35 seconds.

so dirty_background_ratio is working, that's why it took a long time to start writing anything out

Quoted 3 lines

Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.

this setting should do the trick, so there's something going on here that isn't expected.

Quoted 10 lines

So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.

Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.

Quoted 3 lines

So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.

dirty_ratio is the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bug

Quoted 3 lines

Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.

how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?

just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk

Quoted 4 lines

Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??

this should be possible, although it could be considered a bit risky WRT OOM.

NATE

Quoted 87 lines

regards, Anthony

Quoted 22 lines · Nate Diller
Quoted 8 lines · Roger Heflin

I saw it mentioned before that the kernel only allows a certain percentage of total memory to be dirty, I thought the number was around 40%, and I have seen machines with large amounts of ram, hit the 40% and then put the writing application into disk wait until certain amounts of things are written out, and then take it out of disk wait, and repeat when it again hits 40%, given your rate different it would be close to 40% in 50seconds.

yes, on 2.6 there are two tunables which are important here. dirty_background_ratio is the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes active. dirty_ratio changes when the write-throttling code kicks in, which is what Anthony is seeing. The purpose of the write throttling code is to limit the dirtying process to disk bandwidth, so that is a Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100, but you could trigger an OOM if you let it get too high, so maybe try setting it at 85 or so. This should effectively disable the write throttling and give you the bandwidth you want.

Quoted 44 lines

NATE

Quoted 41 lines

And I think that you mean MB(yte) not Mb(it).

Roger

Quoted 26 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony Wesley
Sent: Friday, September 09, 2005 4:11 AM
To: [email protected]
Subject: Re: kernel 2.6.13 buffer strangeness

Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.

regards, Anthony

--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836

Quoted 7 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"

in

Quoted 6 lines
Quoted 4 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#5 lore raw

Okay, just tested a couple of things, here's what I see...

I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop hard drive, only 5400rpm so this is not surprising.

I did this test with my video capture app running, but just displaying data and not writing,

I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's a 4200rpm drive so again not surprising numbers.

The video data is written as individual frames, so the efficiency is a bit below the raw throughput, but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would write its data.

I'd bet a large sum of money that these hard disk figures are correct to within a few percent.

Also, when I am actually capturing I have timed by stopwatch how long the disk activity light is on, and reached about the same conclusion.

Working the problem from the other direction, the only way to explain the early throttling that I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.

regards, Anthony

Quoted 47 lines · Nate Diller
Quoted 3 lines

Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.

this setting should do the trick, so there's something going on here that isn't expected.

Quoted 10 lines

So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.

Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.

Quoted 3 lines

So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.

dirty_ratio is the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bug

Quoted 3 lines

Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.

how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?

just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk

Quoted 4 lines

Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??

this should be possible, although it could be considered a bit risky WRT OOM.

NATE

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#6 lore raw
Quoted 35 lines · Anthony Wesley

Okay, just tested a couple of things, here's what I see...

I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop
hard drive, only 5400rpm so this is not surprising.

I did this test with my video capture app running, but just displaying data and not writing,

I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's
a 4200rpm drive so again not surprising numbers.

The video data is written as individual frames, so the efficiency is a bit below the raw throughput,
but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would
write its data.

I'd bet a large sum of money that these hard disk figures are correct to within a few percent.

Also, when I am actually capturing I have timed by stopwatch how long the disk activity light
is on, and reached about the same conclusion.

Working the problem from the other direction, the only way to explain the early throttling that
I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if
the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds
of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.

Quoted 2 lines

regards, Anthony

i need a vmstat trace ('vmstat 1 > logfile', then run the test, then ^c) before i can comment further. also, your filesystem and scheduler would be interesting to know. you can find out the scheduler with 'cat /sys/block/[drive]/queue/scheduler'.

NATE

Quoted 67 lines
Quoted 9 lines · Nate Diller
Quoted 3 lines

Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.

this setting should do the trick, so there's something going on here that isn't expected.

Quoted 1 line

So as far as I can see there is *no way* to get 3 minutes worth of

buffering

Quoted 3 lines
Quoted 3 lines

by adjusting these parameters.

Just to remind everyone - I have video data coming in at 25MBytes/sec and

I

Quoted 1 line
Quoted 1 line

am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I

want

Quoted 2 lines
Quoted 2 lines

my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I

have

Quoted 1 line
Quoted 1 line

1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes

to

Quoted 14 lines
Quoted 5 lines

consume at 7MBytes/sec.

So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.

dirty_ratio is the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bug

Quoted 3 lines

Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.

Quoted 10 lines

how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?

just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk

Quoted 1 line

Or is this simply not possible with the current kernel I/O setup? Do I

have

Quoted 5 lines
Quoted 3 lines

to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??

this should be possible, although it could be considered a bit risky WRT

OOM.

Quoted 2 lines

NATE

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#7 lore raw

Nate, this test was done with the cfq scheduler, but I've tried them all, and they didn't seem to make any real difference. Let me know if you want me to re-run this with one of the other schedulers. The default scheduler is anticipatory, and that's where the problem first showed up.

Attached is the output of "vmstat -S K 1". I started the capture a couple of seconds after starting this trace and stopped it after 2 minutes.

If I'm interpreting it correctly, this shows a disk throughput averaging about 15-16MBytes/sec.

I saw the video write speed sustain 25Mbytes/sec for about 75 seconds and then it dropped.

dirty_background_ratio is set to 1
dirty_ratio is set to 95

Assuming that these numbers are ok, that indicates a deficit of about 10MBytes/sec between my video in and disk out, so for 1.3Gb of ram as buffer I should see about 130 seconds at full speed before it drops. Instead I see this happen after about 75 seconds.

Thanks again,

Anthony

Quoted 41 lines · Nate Diller
Quoted 34 lines · Anthony Wesley

Okay, just tested a couple of things, here's what I see...

I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop hard drive, only 5400rpm so this is not surprising.

I did this test with my video capture app running, but just displaying data and not writing,

I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's
a 4200rpm drive so again not surprising numbers.

The video data is written as individual frames, so the efficiency is a bit below the raw throughput,
but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would
write its data.

I'd bet a large sum of money that these hard disk figures are correct to within a few percent.

Also, when I am actually capturing I have timed by stopwatch how long the disk activity light
is on, and reached about the same conclusion.

Working the problem from the other direction, the only way to explain the early throttling that
I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if
the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds
of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.

Quoted 4 lines

regards, Anthony

Quoted 121 lines

i need a vmstat trace ('vmstat 1 > logfile', then run the test, then ^c) before i can comment further. also, your filesystem and scheduler would be interesting to know. you can find out the scheduler with 'cat /sys/block/[drive]/queue/scheduler'.

NATE

Quoted 109 lines
Quoted 5 lines · Nate Diller
Quoted 5 lines

Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.

Quoted 9 lines

this setting should do the trick, so there's something going on here that isn't expected.

Quoted 3 lines

So as far as I can see there is *no way* to get 3 minutes worth of

buffering

Quoted 5 lines
Quoted 5 lines

by adjusting these parameters.

Just to remind everyone - I have video data coming in at 25MBytes/sec and

I

Quoted 3 lines
Quoted 3 lines

am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I

want

Quoted 4 lines
Quoted 4 lines

my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I

have

Quoted 3 lines
Quoted 3 lines

1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes

to

Quoted 18 lines
Quoted 7 lines

consume at 7MBytes/sec.

So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.

dirty_ratio is the tubable you want, if it's not working correctly, either there's a problem with your setup, or a bug

Quoted 5 lines

Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.

Quoted 14 lines

how sure are you that you're getting 17MB/s during this test? can you run "vmstat 1" while this is running to verify? which FS and scheduler?

just for interest, what's the raw disk bandwidth (use hdparm, or run a dd, or something)? it would obviously be much better to sustain 25MB/s to disk

Quoted 3 lines

Or is this simply not possible with the current kernel I/O setup? Do I

have

Quoted 8 lines
Quoted 5 lines

to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??

this should be possible, although it could be considered a bit risky WRT

OOM.

Quoted 3 lines

NATE

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 4  0    144 1314396  29440  67444    0    0    16   407  290    17 30 30 38  2
 0  0    144 1314396  29440  67436    0    0     0     0  405  1062 26  4 70  0
 0  0    144 1314388  29440  67436    0    0     0     0  721  1397 27  4 69  0
 0  0    144 1314388  29440  67436    0    0     0     0  732  1554 30  3 67  0
 0  0    144 1300624  29548  80220    0    0    24     0  426  1207 26 13 40 21
 0  0    144 1275444  29744 105148    0    0     8 20752  933  2262 31 21 49  0
 3  2    144 1250396  29912 130076    0    0     4 15286 1004  2357 31 22  0 47
 0  3    144 1225224  30076 155004    0    0     0 14008 1030  2439 28 22  0 49
 1  3    144 1200788  30244 179324    0    0     8 13772 1049  2526 29 21  0 50
 0  3    144 1175616  30412 204256    0    0     4 15964 1102  2696 20 18  0 62
 0  5    144 1150552  30584 229184    0    0     0 15980 1100  2724 21 17  0 62
 2  1    144 1125836  30756 253504    0    0    12 14672 1106  2715 20 19  0 62
 0  7    144 1100788  30920 278432    0    0     0 15380 1079  2665 19 18  0 63
 2  6    144 1075616  31092 303360    0    0     8 15440 1112  2731 21 18  0 61
 0  8    144 1049948  31264 328900    0    0     4 14832 1063  2685 18 22  0 60
 2  8    144 1024776  31436 353828    0    0     0 15620 1117  2815 19 17  0 64
 0  2    144 1001588  31600 376932    0    0    12 15828 1093  2677 18 20  0 62
 2  6    144 976432  31764 401860    0    0     0 14724 1085  2604 21 19  0 60
 0  5    144 950764  31932 427396    0    0     0 15844 1093  2734 21 18  0 61
 1  4    144 924964  32112 452936    0    0    12 15028 1044  2583 22 18  0 61
 0  4    144 899792  32292 477864    0    0     8 14788 1071  2670 18 18  0 64
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  2    144 880200  32432 497320    0    0    12 15228 1103  2676 18 18  0 64
 0  5    144 855152  32596 522248    0    0     0 14604 1053  2598 19 20  0 61
 0  6    144 829980  32760 547176    0    0     0 16004 1124  2741 21 17  0 63
 0  6    144 804188  32932 572716    0    0     4 15896 1081  2704 19 18  0 63
 1  4    144 779116  33112 597644    0    0     8 15020 1094  2722 19 19  0 62
 0 11    144 755672  33272 620840    0    0     4 15876 1076  2750 17 20  0 63
 1 11    144 733104  33416 643244    0    0     0 16220 1110  2795 19 16  0 65
 0  2    144 708056  33580 668172    0    0     0 15836 1074  2668 20 18  0 63
 0  8    144 682884  33744 693104    0    0     0 15952 1115  2758 20 16  0 64
 1  4    144 657216  33920 718640    0    0     0 15908 1115  2765 18 18  0 64
 0  6    144 632664  34088 742960    0    0     8 14928 1080  2663 21 17  0 62
 0  3    144 609972  34252 765456    0    0    16 14888 1055  2606 20 19  0 61
 1  2    144 584320  34432 790992    0    0    12 15044 1094  2709 18 20  0 62
 0  1    144 559768  34600 815316    0    0     8 14948 1069  2634 20 19  0 61
 0  6    144 534596  34772 840244    0    0     0 16188 1092  2645 20 20  0 60
 1  5    144 508928  34944 865780    0    0     4 14832 1063  2625 21 20  0 60
 0  2    144 489972  35080 884628    0    0    12 13700 1057  2515 21 18  0 61
 1  3    144 464800  35256 909556    0    0    12 17008 1101  2728 23 15  0 62
 1  1    144 439008  35432 935096    0    0     8 13692 1109  2708 22 18  0 60
 0  3    144 413960  35604 960024    0    0     0 17072 1096  2676 19 20  0 61
 1  5    144 388804  35776 984952    0    0     8 14484 1082  2640 22 20  0 59
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2    144 366112  35936 1007448    0    0    12 15324 1078  2626 20 21  0 59
 2  2    144 341064  36108 1032376    0    0     8 14544 1060  2551 22 18  0 60
 1  3    144 315272  36276 1057912    0    0     0 15836 1057  2639 22 20  0 59
 0  5    144 290100  36448 1082844    0    0     0 15420 1130  2764 21 19  0 60
 1  5    144 264060  36616 1108384    0    0     0 14836 1078  2648 23 18  0 59
 0  5    144 238640  36780 1133312    0    0     0 16124 1109  2725 20 20  0 60
 2  4    144 215544  36932 1155808    0    0     4 14768 1058  2613 21 18  0 61
 0  9    144 189504  37104 1181344    0    0     4 14064 1125  2823 21 21  0 58
 1  6    144 164084  37276 1206276    0    0     0 16040 1071  2728 19 20  0 61
 1  9    144 138664  37440 1231204    0    0     0 16020 1136  2822 20 20  0 60
 1  1    144 113120  37608 1256132    0    0     4 14604 1070  2734 18 20  0 62
 2  6    144  89560  37764 1279236    0    0     4 14896 1081  2612 17 22  0 61
 0  9    144  64140  37928 1304164    0    0     0 17184 1092  2818 18 21  0 61
 0  9    144  38572  38100 1329096    0    0     0 15784 1126  2833 22 19  0 59
 2  3    144  32628  27604 1345272    0    0     0 14780 1101  2675 21 23  0 56
 1  9    144  32628  16548 1356284    0    0     0 16024 1100  2740 21 22  0 57
 3  9    144  32628  16712 1356028    0    0     4 14852 1069  2729 19 19  0 62
 0  4    144  32628  16880 1355844    0    0     8 15044 1082  2692 19 21  0 60
 1  9    144  32132  17052 1356200    0    0     0 16100 1090  2776 21 20  0 59
 2  6    144  32504  17216 1355784    0    0     0 14712 1097  2719 20 22  0 58
 0  9    144  32628  17384 1355368    0    0     4 14908 1037  2690 22 20  0 58
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  5    144  32380  17536 1355528    0    0    12 13812 1060  2593 24 17  0 59
 0  9    144  32752  17704 1354952    0    0     0 14932 1063  2654 22 23  0 55
 0  9    144  32256  17880 1355244    0    0     0  7228 1113  2881 22 21  0 57
 0  2    144  32008  18044 1355388    0    0     8 14892 1090  2761 23 22  0 56
 1  3    144  32008  18220 1355052    0    0     8 13996 1050  2585 20 22  0 58
 1  9    144  32504  18360 1354524    0    0     8 13252 1021  2530 19 19  0 62
 1  8    144  32876  18516 1353884    0    0     0 15956 1120  2771 22 18  0 60
 1  6    144  32804  18700 1353760    0    0     8 14120 1079  2738 23 22  0 56
 0  8    144  32628  18868 1353748    0    0     4 12808 1221  3006 21 22  0 57
 1  8    144  33052  19024 1353080    0    0     0  6320  989  2631 21 21  0 58
 0  8    144  32756  19192 1353224    0    0     0  6308 1099  2833 21 22  0 57
 0  2    144  32628  19300 1353296    0    0    12 10540 1019  2564 20 18  0 62
 2  8    144  31992  19428 1353644    0    0     4 20276  973  2448 20 19  0 61
 0  8    144  32256  19520 1353376    0    0     4  9816 1103  2694 19 20  0 62
 1  4    144  32028  19600 1353384    0    0     4 18116 1114  2643 19 16  0 65
 1  5    144  33000  19740 1352244    0    0     4 12424 1055  2593 19 21  0 60
 0  8    144  33076  19848 1352068    0    0     4 22344 1111  2682 18 17  0 65
 2  3    144  32480  19964 1352520    0    0    12  9428 1078  2709 18 17  0 65
 1  8    144  32224  20080 1352584    0    0     4 17816 1071  2622 17 21  0 62
 0 11    144  32224  20184 1352604    0    0     4 14052 1054  2643 19 17  0 64
 1 13    144  32052  20292 1352540    0    0     4 15940 1110  2752 21 17  0 63
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1 13    144  32596  20384 1351948    0    0     4 14792 1041  2631 21 14  0 65
 2 14    144  32720  20456 1351652    0    0     4 18476 1077  2676 19 14  0 67
 0  5    144  32720  20604 1351568    0    0     0 13072 1078  2692 21 18  0 61
 0 10    144  32720  20724 1351336    0    0     4 19720 1106  2690 20 18  0 62
 1 13    144  31976  20816 1352016    0    0     4 11964 1065  2639 19 15  0 66
 0 14    144  32596  20876 1351476    0    0     8 19012 1092  2702 19 17  0 65
 1 14    144  31976  21012 1351784    0    0     8 13204 1084  2805 20 19  0 61
 0  2    144  31976  21100 1351776    0    0     0 14896 1078  2706 20 16  0 64
 0  9    144  32476  21240 1351048    0    0     4 18760 1069  2603 20 23  0 58
 1 13    144  32100  21344 1351392    0    0     0 14720 1126  2774 20 17  0 63
 0 14    144  32596  21460 1350860    0    0     4 18992 1095  2735 22 18  0 60
 0 14    144  33096  21576 1350072    0    0     0 14660 1118  2860 22 17  0 61
 1 13    144  31976  21656 1351056    0    0     4 13764 1074  2674 20 18  0 63
 0  4    144  32724  21788 1350176    0    0     0 19236 1089  2710 20 20  0 60
 1  9    144  32720  21888 1350128    0    0     4 12440 1079  2604 20 20  0 61
 0 14    144  32348  21996 1350320    0    0     8 24964 1111  2753 21 15  0 64
 0 13    144  32704  22112 1349948    0    0     4  7960 1088  2791 17 22  0 61
 2 14    144  32208  22196 1350360    0    0     0 22596 1136  2790 20 21  0 60
 0  2    144  32456  22328 1349764    0    0     4  9816 1049  2681 21 19  0 60
 0  8    144  32084  22448 1350124    0    0     0 17024 1105  2668 19 21  0 60
 2 13    144  32084  22544 1350000    0    0     0 19064 1102  2740 18 20  0 62
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  4    144  32828  22668 1349180    0    0     8 18948 1106  2710 18 17  0 65
 1  7    144  31960  22768 1349892    0    0     8 10664 1068  2580 18 21  0 62
 1 12    144  32084  22892 1349488    0    0     0 18428 1118  2753 21 18  0 61
 0 13    144  31960  22996 1349656    0    0     4 14540 1082  2713 19 17  0 64
 0 14    144  32084  23088 1349364    0    0     4 23028 1089  2751 17 17  0 66
 1 13    144  31960  23228 1349220    0    0     0  8720 1103  2785 19 19  0 62
 0 13    144  32580  23320 1348772    0    0     4 14816 1053  2626 20 17  0 64
 0 13    144  32208  23404 1348964    0    0     4 15972 1068  2628 18 16  0 66
 1 14    144  32580  23380 1348432    0    0     0 25216 1106  2808 20 19  0 61
 0 13    144  32704  23144 1348680    0    0     4  7140 1099  2742 17 17  0 66
 0 13    144  32580  22776 1349172    0    0     4 22132 1128  2822 24 15  0 62
 2 14    144  32332  22192 1349820    0    0     0 14152 1062  2691 18 21  0 61
 0 13    144  32208  21628 1350684    0    0     0 15448 1121  2750 18 18  0 64
 1  5    144  31960  20996 1351536    0    0     4 18192 1038  2541 21 21  0 58
 0  9    144  32208  21024 1351088    0    0     0 11980 1119  2689 19 19  0 62
 1 13    144  32828  21100 1350576    0    0     0 19696 1095  2783 21 14  0 65
 0 13    144  32084  21160 1351156    0    0     4 18324 1097  2769 22 16  0 62
 1 13    144  32832  21276 1350352    0    0     4 10688 1076  2709 20 18  0 62
 0 13    144  32952  21388 1350604    0    0     4 20564 1124  2843 22 18  0 60
 2  1    144  32952  21392 1350604    0    0     4 14460 1146  2729 17 15  0 68
 0  1    144  32952  21404 1350604    0    0     4 16548 1402  3150 19 12  0 69
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1    144  32952  21408 1350604    0    0     4 18056 1167  2657 16 14  0 70
 0  1    144  32952  21408 1350604    0    0     0 18548 1189  2675 17 17  0 67
 0  1    144  32952  21412 1350604    0    0     4 14948 1067  2490 17 12  0 71

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 1 line · Nate Diller

just found the culprit. guess i should have read the code the first time. get_dirty_limits() in drivers/block/page_writeback.c has a hard-coded upper limit to dirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio > unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you can have all the OOM goodness you want.

Excellent. OOM here I come.

Quoted 1 line

i really recommend you focus on getting better disk bandwidth, you stand to gain a lot more from that approach. i presume you're on ext3; perhaps you should try reiser4 or xfs, they are more likely to meet your disk bandwidth requirements.

Yep, pursuing this as well, also looking to add more RAM to the machine. At this stage I was just trying to understand the numbers that I was seeing, so I could work out the best way to proceed.

I'm using ext2 at the moment, on the assumption that the journal would cost me a bit of performance so I left it out :-)

I'll certainly try the other filesystems as you suggest.

Thanks again, Anthony

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#10 lore raw

Re: kernel 2.6.13 buffer strangeness - FIXED

In reply to a message not in the archive
Quoted 1 line · Nate Diller

just found the culprit. guess i should have read the code the first time. get_dirty_limits() in drivers/block/page_writeback.c has a hard-coded upper limit to dirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio > unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you can have all the OOM goodness you want.

Well well well.

Thank you Nate!

I changed that bit of code to:

	if (dirty_ratio > unmapped_ratio - 10) 
		dirty_ratio = unmapped_ratio - 10;

and added a couple of sanity checks so that it couldn't get below 5 or above 95.

Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.

Guess what? Now my video streams for 120 seconds before being throttled. This is *much better*. It's now quite feasible to reach 30 seconds by finding a faster disk and/or adding another 512M or RAM.

I guess the hard-coded limit is in there for a reason, but I sure wish it was adjustable without this sort of hacking.

regards, Anthony

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#11 lore raw

Re: kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison

In reply to a message not in the archive
Quoted 3 lines · Nate Diller

i really recommend you focus on getting better disk bandwidth, you stand to gain a lot more from that approach. i presume you're on ext3; perhaps you should try reiser4 or xfs, they are more likely to meet your disk bandwidth requirements.

NATE

While I have already solved the issue that was troubling me, I also spent some time comparing different filesystems as reccommended by Nate, with interesting results.

My method was simple - make a filesystem and the set it as the target for my video capture. With video coming in at 25MBytes/sec and going out to disk at about 15-20MBytes/sec it is an interesting test of the vm and filesystem.

I compared ext2,ext3,xfs,vfat,reiser and reiser4.

The hands-down winner was ext2. All the others showed problems of either lower disk throughput or dropped frames during video capture.

Only ext2 went the full distance - no dropped frames until we run out of RAM, and good disk throughput.

xfs,reiser and reiser4 had slightly higher disk write speed, but showed performance problems that caused lots of dropped frames so they must be ruled out at this stage.

I know that xfs and reiser4 are supposed to be faster for some things, but it seems to me that they are not the best choice when you are predominantly writing lots and lots of 600k files :-)

regards, Anthony

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#12 lore raw

Re: kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison

Quoted 5 lines · Anthony Wesley

I compared ext2,ext3,xfs,vfat,reiser and reiser4.

The hands-down winner was ext2. All the others showed problems of either lower disk throughput or dropped frames during video capture.

ext2 is a good filesystem. For that sort of application all the journaling gunk can really get in the way.

You should have tested ext3 with data=writeback.

#13 lore raw

Re: kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison

Quoted 4 lines · Andrew Morton
Quoted 2 lines · Anthony Wesley

I compared ext2,ext3,xfs,vfat,reiser and reiser4.

Quoted 9 lines
Quoted 2 lines

The hands-down winner was ext2. All the others showed problems of either lower disk throughput or dropped frames during video capture.

ext2 is a good filesystem. For that sort of application all the journaling gunk can really get in the way.

You should have tested ext3 with data=writeback.

Ask and ye shall receive...

I created an ext3 fs, mounted it with data=writeback and gave it a quick spin.

The result? Lots of pauses and dropped frames during capture. This is during the part of the process where I have gobs of free RAM that's being used for buffering so dropping frames here is a cardinal sin.

Dunno why it's happening, but I saw it also with xfs and reiser4. ext2 on the other hand chugs along happily, no pauses, no dropped frames until we run out of free RAM (takes about 2 minutes now after the simple kernel change).

I can understand dropped frames after we run out of ram, but not before.

regards, Anthony

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#14 lore raw
In reply to Nate Diller

Based on a better understanding of dirty_ratio and dirty_background ratio (thanks Nate) I just tried the following test:

dirty_ratio set to 95
dirty_background_ratio set to 1

Quoted 1 line

From Nate's description of these parameters, this should mean that the disk writes

start almost immediately, and the kernel will allow 95% of RAM to become dirty before applying the throttle.

Ok, so with 25Mbytes/s coming in, and 17Mbytes/sec going out to disk, the dirty pages should be growing at 7Mbytes/sec. With these parameters set as above I should see about 3 minutes of full speed video before the throttle is applied since I have about 1.3Gb of RAM free for buffering..

*But* when I try this experiment I hit the throttle after only 65 seconds - an improvement to be sure, but still a long way short of the 180 seconds that it ought to take.

Part of the test works as expected - the disk writes begin almost immediately due to the low value for dirty_background_ratio, but the rest is a mystery.

It really looks as if the pages aren't being marked as clean fast enough after they're written.

How else can it take only 70 seconds to reach 95% dirty when I have 1.3Gb of available RAM and data coming in at 25MBytes/sec and out at 17MBytes/sec? It doesn't make any sense...

regards, Anthony

Quoted 11 lines · Nate Diller

yes, on 2.6 there are two tunables which are important here. dirty_background_ratio is the threshold where the kernel will begin flushing dirty buffers, so it should change how soon the disk becomes
active. dirty_ratio changes when the write-throttling code kicks in,
which is what Anthony is seeing. The purpose of the write throttling
code is to limit the dirtying process to disk bandwidth, so that is a
Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100,
but you could trigger an OOM if you let it get too high, so maybe try
setting it at 85 or so. This should effectively disable the write
throttling and give you the bandwidth you want.

NATE

Signature · 7 lines
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

[email protected]
Phone: (02) 62595404 or 0419409836
#15 lore raw
Quoted 2 lines · Anthony Wesley

How else can it take only 70 seconds to reach 95% dirty when I have 1.3Gb of available RAM and data coming in at 25MBytes/sec and out at 17MBytes/sec? It doesn't make any sense...

What architecture? x86?

If so then bear in mind that your memory is split into 500MB highmem and 800MB lowmem. The kernel might be starting I/O due to the highmem zone being full of dirty pages. That'd be wrong of it if so - it's supposed to just fall back to lowmem for the page allocations, but that code has changed quite a bit in the two years since I got all that working...

You need to run `watch -n1 cat /proc/meminfo' while doing these tests...