Cache flush question.
Subsystems (2)
Hi anyone.
If somebody's got a sec to enlighten me, there's some phenomenon I recently came across and found somewhat counterintuitive first.
Whenever I
1. Dirty a bunch of pages backed by an NFS mount on some server.
2. Block the traffic with iptables (TCP, assuming that mattered). Still plenty of writeback pending.
3. Sync
I see #3 drive the dirty count in /proc/meminfo drop back to almost-zero, immediately. The sync itself blocks, though.
So the pages are called clean the moment the write got queued, not acked? Leaving the rest just to retransmits by the socket then? Is this just done so because one can, or would that order rather matter for consistency?
Thanks,
Daniel
Quoted 15 lines · Daniel Stodden
Hi anyone.
If somebody's got a sec to enlighten me, there's some phenomenon I recently came across and found somewhat counterintuitive first.
Whenever I
1. Dirty a bunch of pages backed by an NFS mount on some server.
2. Block the traffic with iptables (TCP, assuming that mattered). Still plenty of writeback pending.
3. Sync
Quoted 7 lines
I see #3 drive the dirty count in
/proc/meminfodrop back to almost-zero, immediately. The sync itself blocks, though.So the pages are called clean the moment the write got queued, not acked? Leaving the rest just to retransmits by the socket then? Is this just done so because one can, or would that order rather matter for consistency?
At a wild guess, maybe you're experiencing what Jim Gettys dubbed "buffer bloat".
http://lwn.net/Articles/419714/
Specficially, does ifconfig show a txqueuelen of 1000 for your device? That means the device is buffering 1000 outbound packets, for no readily apparent reason (other than to screw up latency). With an MTU of 1500 that's a megabyte and a half of outgoing data constipated in the network layer.
NFS also has some cacheing of its own I don't understand yet, for "non-idempotent" transactions. Described in this OLS paper:
http://kernel.org/doc/ols/2006/ols2006v2-pages-59-72.pdf
Rob
Quoted 14 lines · Daniel Stodden
Hi anyone.
If somebody's got a sec to enlighten me, there's some phenomenon I recently came across and found somewhat counterintuitive first.
Whenever I
1. Dirty a bunch of pages backed by an NFS mount on some server.
2. Block the traffic with iptables (TCP, assuming that mattered). Still plenty of writeback pending.
3. Sync
Quoted 7 lines
I see #3 drive the dirty count in
/proc/meminfodrop back to almost-zero, immediately. The sync itself blocks, though.So the pages are called clean the moment the write got queued, not acked? Leaving the rest just to retransmits by the socket then? Is this just done so because one can, or would that order rather matter for consistency?
Take a look at the 'Writeback:' count, which should turn non-zero when you hit #3.
The VM allows pages to be either dirty or in writeback, but not both at the same time. This is not NFS-specific. The same rule applies to local filesystems.
Cheers
Trond
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 23 lines · Trond Myklebust
Quoted 14 lines · Daniel Stodden
Hi anyone.
If somebody's got a sec to enlighten me, there's some phenomenon I recently came across and found somewhat counterintuitive first.
Whenever I
1. Dirty a bunch of pages backed by an NFS mount on some server.
2. Block the traffic with iptables (TCP, assuming that mattered). Still plenty of writeback pending.
3. Sync
Quoted 7 lines
I see #3 drive the dirty count in
/proc/meminfodrop back to almost-zero, immediately. The sync itself blocks, though.So the pages are called clean the moment the write got queued, not acked? Leaving the rest just to retransmits by the socket then? Is this just done so because one can, or would that order rather matter for consistency?
Quoted 6 lines
Take a look at the 'Writeback:' count, which should turn non-zero when you hit #3.
The VM allows pages to be either dirty or in writeback, but not both at the same time. This is not NFS-specific. The same rule applies to local filesystems.
Ah. That explains everything. Actually a question then, thanks for the clarification :)
Rob Landley's comment regarding tx queue size somewhat made a good point too. But, given the rates I see, this queues mostly cache pages on the transport, not copies, right?
Thanks.
Daniel
Quoted 31 lines · Daniel Stodden
Quoted 23 lines · Trond Myklebust
Quoted 14 lines · Daniel Stodden
Hi anyone.
If somebody's got a sec to enlighten me, there's some phenomenon I recently came across and found somewhat counterintuitive first.
Whenever I
1. Dirty a bunch of pages backed by an NFS mount on some server.
2. Block the traffic with iptables (TCP, assuming that mattered). Still plenty of writeback pending.
3. Sync
Quoted 7 lines
I see #3 drive the dirty count in
/proc/meminfodrop back to almost-zero, immediately. The sync itself blocks, though.So the pages are called clean the moment the write got queued, not acked? Leaving the rest just to retransmits by the socket then? Is this just done so because one can, or would that order rather matter for consistency?
Quoted 6 lines
Take a look at the 'Writeback:' count, which should turn non-zero when you hit #3.
The VM allows pages to be either dirty or in writeback, but not both at the same time. This is not NFS-specific. The same rule applies to local filesystems.
Quoted 5 lines
Ah. That explains everything. Actually a question then, thanks for the clarification :)
Rob Landley's comment regarding tx queue size somewhat made a good point too.
Not nearly as good as I thought at the time.
Quoted 2 lines
But, given the rates I see, this queues mostly cache pages on the transport, not copies, right?
Easy enough to tell: there's a "Writeback" field in /proc/meminfo. Add 'em up and see what's missing.
Rob