extremely slow nfs when sync enabled
Subsystems (2)
I've been observing some very slow nfs write performance when the server has `sync' in /etc/exports
I want to avoid using async, but I have tested it and on my gigabit network, it gives almost the same speed as if I was on the server itself. (e.g. 30MB/sec to one disk, or less than 1MB/sec to the same disk over NFS with `sync')
I'm using Debian 6 with 2.6.38 kernels on client and server, NFSv3
I've also tried a client running Debian 7/Linux 3.2.0 with both NFSv3 and NFSv4, speed is still slow
Looking at iostat on the server, I notice that avgrq-sz = 8 sectors (4096 bytes) throughout the write operations
I've tried various tests, e.g. dd a large file, or unpack a tarball with many small files, the iostat output is always the same
Looking at /proc/mounts on the clients, everything looks good, large wsize, tcp:
rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.x.x.x,mountvers=3,mountport=58727,mountproto=udp,local_lock=none,addr=192.x.x.x 0 0
and
rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.x.x.x.,minorversion=0,local_lock=none,addr=192.x.x.x 0 0
and in /proc/fs/nfs/exports on the server, I have sync and wdelay:
/nfs4/daniel 192.168.1.0/24,192.x.x.x(rw,insecure,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9,sec=1) /home/daniel 192.168.1.0/24,192.x.x.x(rw,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9)
Can anyone suggest anything else? Or is this really the performance hit of `sync'?
T24gU3VuLCAyMDEyLTA1LTA2IGF0IDAzOjAwICswMDAwLCBEYW5pZWwgUG9jb2NrIHdyb3RlOg0K PiANCj4gSSd2ZSBiZWVuIG9ic2VydmluZyBzb21lIHZlcnkgc2xvdyBuZnMgd3JpdGUgcGVyZm9y bWFuY2Ugd2hlbiB0aGUgc2VydmVyDQo+IGhhcyBgc3luYycgaW4gL2V0Yy9leHBvcnRzDQo+IA0K PiBJIHdhbnQgdG8gYXZvaWQgdXNpbmcgYXN5bmMsIGJ1dCBJIGhhdmUgdGVzdGVkIGl0IGFuZCBv biBteSBnaWdhYml0DQo+IG5ldHdvcmssIGl0IGdpdmVzIGFsbW9zdCB0aGUgc2FtZSBzcGVlZCBh cyBpZiBJIHdhcyBvbiB0aGUgc2VydmVyDQo+IGl0c2VsZi4gKGUuZy4gMzBNQi9zZWMgdG8gb25l IGRpc2ssIG9yIGxlc3MgdGhhbiAxTUIvc2VjIHRvIHRoZSBzYW1lDQo+IGRpc2sgb3ZlciBORlMg d2l0aCBgc3luYycpDQo+IA0KPiBJJ20gdXNpbmcgRGViaWFuIDYgd2l0aCAyLjYuMzgga2VybmVs cyBvbiBjbGllbnQgYW5kIHNlcnZlciwgTkZTdjMNCj4gDQo+IEkndmUgYWxzbyB0cmllZCBhIGNs aWVudCBydW5uaW5nIERlYmlhbiA3L0xpbnV4IDMuMi4wIHdpdGggYm90aCBORlN2Mw0KPiBhbmQg TkZTdjQsIHNwZWVkIGlzIHN0aWxsIHNsb3cNCj4gDQo+IExvb2tpbmcgYXQgaW9zdGF0IG9uIHRo ZSBzZXJ2ZXIsIEkgbm90aWNlIHRoYXQgYXZncnEtc3ogPSA4IHNlY3RvcnMNCj4gKDQwOTYgYnl0 ZXMpIHRocm91Z2hvdXQgdGhlIHdyaXRlIG9wZXJhdGlvbnMNCj4gDQo+IEkndmUgdHJpZWQgdmFy aW91cyB0ZXN0cywgZS5nLiBkZCBhIGxhcmdlIGZpbGUsIG9yIHVucGFjayBhIHRhcmJhbGwgd2l0 aA0KPiBtYW55IHNtYWxsIGZpbGVzLCB0aGUgaW9zdGF0IG91dHB1dCBpcyBhbHdheXMgdGhlIHNh bWUNCg0KV2VyZSB5b3UgdXNpbmcgJ2NvbnY9c3luYyc/DQoNCj4gTG9va2luZyBhdCAvcHJvYy9t b3VudHMgb24gdGhlIGNsaWVudHMsIGV2ZXJ5dGhpbmcgbG9va3MgZ29vZCwgbGFyZ2UNCj4gd3Np emUsIHRjcDoNCj4gDQo+IHJ3LHJlbGF0aW1lLHZlcnM9Myxyc2l6ZT0xMDQ4NTc2LHdzaXplPTEw NDg1NzYsbmFtbGVuPTI1NSxoYXJkLHByb3RvPXRjcCx0aW1lbz02MDAscmV0cmFucz0yLHNlYz1z eXMsbW91bnRhZGRyPTE5Mi54LngueCxtb3VudHZlcnM9Myxtb3VudHBvcnQ9NTg3MjcsbW91bnRw cm90bz11ZHAsbG9jYWxfbG9jaz1ub25lLGFkZHI9MTkyLngueC54DQo+IDAgMA0KPiANCj4gYW5k DQo+ICBydyxyZWxhdGltZSx2ZXJzPTQscnNpemU9MTA0ODU3Nix3c2l6ZT0xMDQ4NTc2LG5hbWxl bj0yNTUsaGFyZCxwcm90bz10Y3AscG9ydD0wLHRpbWVvPTYwMCxyZXRyYW5zPTIsc2VjPXN5cyxj bGllbnRhZGRyPTE5Mi54LngueC4sbWlub3J2ZXJzaW9uPTAsbG9jYWxfbG9jaz1ub25lLGFkZHI9 MTkyLngueC54IDAgMA0KPiANCj4gYW5kIGluIC9wcm9jL2ZzL25mcy9leHBvcnRzIG9uIHRoZSBz ZXJ2ZXIsIEkgaGF2ZSBzeW5jIGFuZCB3ZGVsYXk6DQo+IA0KPiAvbmZzNC9kYW5pZWwNCj4gMTky LjE2OC4xLjAvMjQsMTkyLngueC54KHJ3LGluc2VjdXJlLHJvb3Rfc3F1YXNoLHN5bmMsd2RlbGF5 LG5vX3N1YnRyZWVfY2hlY2ssdXVpZD1hYTJhNmYzNzo5Y2M5NGVlYjpiY2JmOTgzYzpkNmUwNDFk OSxzZWM9MSkNCj4gL2hvbWUvZGFuaWVsDQo+IDE5Mi4xNjguMS4wLzI0LDE5Mi54LngueChydyxy b290X3NxdWFzaCxzeW5jLHdkZWxheSxub19zdWJ0cmVlX2NoZWNrLHV1aWQ9YWEyYTZmMzc6OWNj OTRlZWI6YmNiZjk4M2M6ZDZlMDQxZDkpDQo+IA0KPiBDYW4gYW55b25lIHN1Z2dlc3QgYW55dGhp bmcgZWxzZT8gIE9yIGlzIHRoaXMgcmVhbGx5IHRoZSBwZXJmb3JtYW5jZSBoaXQNCj4gb2YgYHN5 bmMnPw0KDQpJdCByZWFsbHkgZGVwZW5kcyBvbiB5b3VyIGRpc2sgc2V0dXAuIFBhcnRpY3VsYXJs eSB3aGVuIHlvdXIgZmlsZXN5c3RlbQ0KaXMgdXNpbmcgYmFycmllcnMgKGVuYWJsZWQgYnkgZGVm YXVsdCBvbiBleHQ0IGFuZCB4ZnMpLCBhIGxvdCBvZiByYWlkDQpzZXR1cHMgcmVhbGx5IF9zdWNr XyBhdCBkZWFsaW5nIHdpdGggZnN5bmMoKS4gVGhlIGxhdHRlciBpcyB1c2VkIGV2ZXJ5DQp0aW1l IHRoZSBORlMgY2xpZW50IHNlbmRzIGEgQ09NTUlUIG9yIHRydW5jKCkgaW5zdHJ1Y3Rpb24sIGFu ZCBmb3INCnByZXR0eSBtdWNoIGFsbCBmaWxlIGFuZCBkaXJlY3RvcnkgY3JlYXRpb24gb3BlcmF0 aW9ucyAoeW91IGNhbiB1c2UNCiduZnNzdGF0JyB0byBtb25pdG9yIGhvdyBtYW55IHN1Y2ggb3Bl cmF0aW9ucyB0aGUgTkZTIGNsaWVudCBpcyBzZW5kaW5nDQphcyBwYXJ0IG9mIHlvdXIgdGVzdCku DQoNCkxvY2FsIGRpc2sgY2FuIGdldCBhd2F5IHdpdGggZG9pbmcgYSBsb3QgbGVzcyBmc3luYygp LCBiZWNhdXNlIHRoZSBjYWNoZQ0KY29uc2lzdGVuY3kgZ3VhcmFudGVlcyBhcmUgZGlmZmVyZW50 Og0KICAgICAgKiBpbiBORlMsIHRoZSBzZXJ2ZXIgaXMgYWxsb3dlZCB0byBjcmFzaCBvciByZWJv b3Qgd2l0aG91dA0KICAgICAgICBhZmZlY3RpbmcgdGhlIGNsaWVudCdzIHZpZXcgb2YgdGhlIGZp bGVzeXN0ZW0uDQogICAgICAqIGluIHRoZSBsb2NhbCBmaWxlIHN5c3RlbSwgdGhlIGV4cGVjdGF0 aW9uIGlzIHRoYXQgb24gcmVib290IGFueQ0KICAgICAgICBkYXRhIGxvc3QgaXMgd29uJ3QgbmVl ZCB0byBiZSByZWNvdmVyZWQgKHRoZSBhcHBsaWNhdGlvbiB3aWxsDQogICAgICAgIGhhdmUgdXNl ZCBmc3luYygpIGZvciBhbnkgZGF0YSB0aGF0IGRvZXMgbmVlZCB0byBiZSBwZXJzaXN0ZW50KS4N CiAgICAgICAgT25seSB0aGUgZGlzayBmaWxlc3lzdGVtIHN0cnVjdHVyZXMgbmVlZCB0byBiZSBy ZWNvdmVyZWQsIGFuZA0KICAgICAgICB0aGF0IGlzIGRvbmUgdXNpbmcgdGhlIGpvdXJuYWwgKG9y IGZzY2spLg0KDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50 YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5j b20NCg0K
Quoted 15 lines · Trond
Quoted 14 lines · Daniel Pocock
I've been observing some very slow nfs write performance when the server has `sync' in
/etc/exportsI want to avoid using async, but I have tested it and on my gigabit network, it gives almost the same speed as if I was on the server itself. (e.g. 30MB/sec to one disk, or less than 1MB/sec to the same disk over NFS with `sync')
I'm using Debian 6 with 2.6.38 kernels on client and server, NFSv3
I've also tried a client running Debian 7/Linux 3.2.0 with both NFSv3 and NFSv4, speed is still slow
Quoted 7 lines
Quoted 5 lines
Looking at iostat on the server, I notice that avgrq-sz = 8 sectors (4096 bytes) throughout the write operations
I've tried various tests, e.g. dd a large file, or unpack a tarball with many small files, the iostat output is always the same
Were you using 'conv=sync'?
No, it was not using conv=sync, just the vanilla dd:
dd if=/dev/zero of=some-fat-file bs=65536 count=65536
Quoted 16 lines
Quoted 16 lines
Looking at
/proc/mountson the clients, everything looks good, large wsize, tcp:rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.x.x.x,mountvers=3,mountport=58727,mountproto=udp,local_lock=none,addr=192.x.x.x 0 0
and
rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.x.x.x.,minorversion=0,local_lock=none,addr=192.x.x.x 0 0and in
/proc/fs/nfs/exportson the server, I have sync and wdelay:/nfs4/daniel 192.168.1.0/24,192.x.x.x(rw,insecure,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9,sec=1)
/home/daniel192.168.1.0/24,192.x.x.x(rw,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9)
Quoted 5 lines
Quoted 2 lines
Can anyone suggest anything else? Or is this really the performance hit of `sync'?
It really depends on your disk setup. Particularly when your filesystem is using barriers (enabled by default on ext4 and xfs), a lot of raid
On the server, I've tried both ext3 and ext4, explicitly changing things like data=writeback,barrier=0, but the problem remains
The only thing that made it faster was using hdparm -W1 /dev/sd[ab] to enable the write-back cache on the disk
Quoted 1 line
setups really _suck_ at dealing with
fsync(). The latter is used every
I'm using md RAID1, my setup is like this:
2x 1TB SATA disks ST31000528AS (7200rpm with 32MB cache and NCQ)
SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)
- not using any of the BIOS softraid stuff
Both devices have identical partitioning:
1. 128MB boot
2. md volume (1TB - 128MB)
The entire md volume (/dev/md2) is then used as a PV for LVM
I do my write tests on a fresh LV with no fragmentation
Quoted 4 lines
time the NFS client sends a COMMIT or
trunc()instruction, and for pretty much all file and directory creation operations (you can use 'nfsstat' to monitor how many such operations the NFS client is sending as part of your test).
I know that my two tests are very different in that way:
- dd is just writing one big file, no fsync
- unpacking a tarball (or compiling a large C++ project) does a lot of small writes with many fsyncs
In both cases, it is slow
Quoted 9 lines
Local disk can get away with doing a lot less
fsync(), because the cache consistency guarantees are different:
* in NFS, the server is allowed to crash or reboot without affecting the client's view of the filesystem.
* in the local file system, the expectation is that on reboot any data lost is won't need to be recovered (the application will have usedfsync()for any data that does need to be persistent). Only the disk filesystem structures need to be recovered, and that is done using the journal (or fsck).
Is this an intractable problem though?
Or do people just work around this, for example, enable async and write-back cache, and then try to manage the risk by adding a UPS and/or battery backed cache to their RAID setup (to reduce the probability of unclean shutdown)?
T24gU3VuLCAyMDEyLTA1LTA2IGF0IDIxOjIzICswMDAwLCBEYW5pZWwgUG9jb2NrIHdyb3RlOg0K PiANCj4gT24gMDYvMDUvMTIgMTg6MjMsIE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+ID4gT24g U3VuLCAyMDEyLTA1LTA2IGF0IDAzOjAwICswMDAwLCBEYW5pZWwgUG9jb2NrIHdyb3RlOg0KPiA+ Pg0KPiA+PiBJJ3ZlIGJlZW4gb2JzZXJ2aW5nIHNvbWUgdmVyeSBzbG93IG5mcyB3cml0ZSBwZXJm b3JtYW5jZSB3aGVuIHRoZSBzZXJ2ZXINCj4gPj4gaGFzIGBzeW5jJyBpbiAvZXRjL2V4cG9ydHMN Cj4gPj4NCj4gPj4gSSB3YW50IHRvIGF2b2lkIHVzaW5nIGFzeW5jLCBidXQgSSBoYXZlIHRlc3Rl ZCBpdCBhbmQgb24gbXkgZ2lnYWJpdA0KPiA+PiBuZXR3b3JrLCBpdCBnaXZlcyBhbG1vc3QgdGhl IHNhbWUgc3BlZWQgYXMgaWYgSSB3YXMgb24gdGhlIHNlcnZlcg0KPiA+PiBpdHNlbGYuIChlLmcu IDMwTUIvc2VjIHRvIG9uZSBkaXNrLCBvciBsZXNzIHRoYW4gMU1CL3NlYyB0byB0aGUgc2FtZQ0K PiA+PiBkaXNrIG92ZXIgTkZTIHdpdGggYHN5bmMnKQ0KPiA+Pg0KPiA+PiBJJ20gdXNpbmcgRGVi aWFuIDYgd2l0aCAyLjYuMzgga2VybmVscyBvbiBjbGllbnQgYW5kIHNlcnZlciwgTkZTdjMNCj4g Pj4NCj4gPj4gSSd2ZSBhbHNvIHRyaWVkIGEgY2xpZW50IHJ1bm5pbmcgRGViaWFuIDcvTGludXgg My4yLjAgd2l0aCBib3RoIE5GU3YzDQo+ID4+IGFuZCBORlN2NCwgc3BlZWQgaXMgc3RpbGwgc2xv dw0KPiA+Pg0KPiA+PiBMb29raW5nIGF0IGlvc3RhdCBvbiB0aGUgc2VydmVyLCBJIG5vdGljZSB0 aGF0IGF2Z3JxLXN6ID0gOCBzZWN0b3JzDQo+ID4+ICg0MDk2IGJ5dGVzKSB0aHJvdWdob3V0IHRo ZSB3cml0ZSBvcGVyYXRpb25zDQo+ID4+DQo+ID4+IEkndmUgdHJpZWQgdmFyaW91cyB0ZXN0cywg ZS5nLiBkZCBhIGxhcmdlIGZpbGUsIG9yIHVucGFjayBhIHRhcmJhbGwgd2l0aA0KPiA+PiBtYW55 IHNtYWxsIGZpbGVzLCB0aGUgaW9zdGF0IG91dHB1dCBpcyBhbHdheXMgdGhlIHNhbWUNCj4gPiAN Cj4gPiBXZXJlIHlvdSB1c2luZyAnY29udj1zeW5jJz8NCj4gDQo+IE5vLCBpdCB3YXMgbm90IHVz aW5nIGNvbnY9c3luYywganVzdCB0aGUgdmFuaWxsYSBkZDoNCj4gDQo+IGRkIGlmPS9kZXYvemVy byBvZj1zb21lLWZhdC1maWxlIGJzPTY1NTM2IGNvdW50PTY1NTM2DQoNClRoZW4gdGhlIHJlc3Vs dHMgYXJlIG5vdCBjb21wYXJhYmxlLg0KDQo+ID4+IExvb2tpbmcgYXQgL3Byb2MvbW91bnRzIG9u IHRoZSBjbGllbnRzLCBldmVyeXRoaW5nIGxvb2tzIGdvb2QsIGxhcmdlDQo+ID4+IHdzaXplLCB0 Y3A6DQo+ID4+DQo+ID4+IHJ3LHJlbGF0aW1lLHZlcnM9Myxyc2l6ZT0xMDQ4NTc2LHdzaXplPTEw NDg1NzYsbmFtbGVuPTI1NSxoYXJkLHByb3RvPXRjcCx0aW1lbz02MDAscmV0cmFucz0yLHNlYz1z eXMsbW91bnRhZGRyPTE5Mi54LngueCxtb3VudHZlcnM9Myxtb3VudHBvcnQ9NTg3MjcsbW91bnRw cm90bz11ZHAsbG9jYWxfbG9jaz1ub25lLGFkZHI9MTkyLngueC54DQo+ID4+IDAgMA0KPiA+Pg0K PiA+PiBhbmQNCj4gPj4gIHJ3LHJlbGF0aW1lLHZlcnM9NCxyc2l6ZT0xMDQ4NTc2LHdzaXplPTEw NDg1NzYsbmFtbGVuPTI1NSxoYXJkLHByb3RvPXRjcCxwb3J0PTAsdGltZW89NjAwLHJldHJhbnM9 MixzZWM9c3lzLGNsaWVudGFkZHI9MTkyLngueC54LixtaW5vcnZlcnNpb249MCxsb2NhbF9sb2Nr PW5vbmUsYWRkcj0xOTIueC54LnggMCAwDQo+ID4+DQo+ID4+IGFuZCBpbiAvcHJvYy9mcy9uZnMv ZXhwb3J0cyBvbiB0aGUgc2VydmVyLCBJIGhhdmUgc3luYyBhbmQgd2RlbGF5Og0KPiA+Pg0KPiA+ PiAvbmZzNC9kYW5pZWwNCj4gPj4gMTkyLjE2OC4xLjAvMjQsMTkyLngueC54KHJ3LGluc2VjdXJl LHJvb3Rfc3F1YXNoLHN5bmMsd2RlbGF5LG5vX3N1YnRyZWVfY2hlY2ssdXVpZD1hYTJhNmYzNzo5 Y2M5NGVlYjpiY2JmOTgzYzpkNmUwNDFkOSxzZWM9MSkNCj4gPj4gL2hvbWUvZGFuaWVsDQo+ID4+ IDE5Mi4xNjguMS4wLzI0LDE5Mi54LngueChydyxyb290X3NxdWFzaCxzeW5jLHdkZWxheSxub19z dWJ0cmVlX2NoZWNrLHV1aWQ9YWEyYTZmMzc6OWNjOTRlZWI6YmNiZjk4M2M6ZDZlMDQxZDkpDQo+ ID4+DQo+ID4+IENhbiBhbnlvbmUgc3VnZ2VzdCBhbnl0aGluZyBlbHNlPyAgT3IgaXMgdGhpcyBy ZWFsbHkgdGhlIHBlcmZvcm1hbmNlIGhpdA0KPiA+PiBvZiBgc3luYyc/DQo+ID4gDQo+ID4gSXQg cmVhbGx5IGRlcGVuZHMgb24geW91ciBkaXNrIHNldHVwLiBQYXJ0aWN1bGFybHkgd2hlbiB5b3Vy IGZpbGVzeXN0ZW0NCj4gPiBpcyB1c2luZyBiYXJyaWVycyAoZW5hYmxlZCBieSBkZWZhdWx0IG9u IGV4dDQgYW5kIHhmcyksIGEgbG90IG9mIHJhaWQNCj4gDQo+IE9uIHRoZSBzZXJ2ZXIsIEkndmUg dHJpZWQgYm90aCBleHQzIGFuZCBleHQ0LCBleHBsaWNpdGx5IGNoYW5naW5nIHRoaW5ncw0KPiBs aWtlIGRhdGE9d3JpdGViYWNrLGJhcnJpZXI9MCwgYnV0IHRoZSBwcm9ibGVtIHJlbWFpbnMNCj4g DQo+IFRoZSBvbmx5IHRoaW5nIHRoYXQgbWFkZSBpdCBmYXN0ZXIgd2FzIHVzaW5nIGhkcGFybSAt VzEgL2Rldi9zZFthYl0gdG8NCj4gZW5hYmxlIHRoZSB3cml0ZS1iYWNrIGNhY2hlIG9uIHRoZSBk aXNrDQoNClRoYXQgc2hvdWxkIGluIHByaW5jaXBsZSBiZSBzYWZlIHRvIGRvIGFzIGxvbmcgYXMg eW91IGFyZSB1c2luZw0KYmFycmllcj0xLg0KDQo+ID4gc2V0dXBzIHJlYWxseSBfc3Vja18gYXQg ZGVhbGluZyB3aXRoIGZzeW5jKCkuIFRoZSBsYXR0ZXIgaXMgdXNlZCBldmVyeQ0KPiANCj4gSSdt IHVzaW5nIG1kIFJBSUQxLCBteSBzZXR1cCBpcyBsaWtlIHRoaXM6DQo+IA0KPiAyeCAxVEIgU0FU QSBkaXNrcyBTVDMxMDAwNTI4QVMgKDcyMDBycG0gd2l0aCAzMk1CIGNhY2hlIGFuZCBOQ1EpDQo+ IA0KPiBTQVRBIGNvbnRyb2xsZXI6IEFUSSBUZWNobm9sb2dpZXMgSW5jIFNCNzAwL1NCODAwIFNB VEEgQ29udHJvbGxlciBbQUhDSQ0KPiBtb2RlXSAocmV2IDQwKQ0KPiAtIG5vdCB1c2luZyBhbnkg b2YgdGhlIEJJT1Mgc29mdHJhaWQgc3R1ZmYNCj4gDQo+IEJvdGggZGV2aWNlcyBoYXZlIGlkZW50 aWNhbCBwYXJ0aXRpb25pbmc6DQo+IDEuIDEyOE1CIGJvb3QNCj4gMi4gbWQgdm9sdW1lICgxVEIg LSAxMjhNQikNCj4gDQo+IFRoZSBlbnRpcmUgbWQgdm9sdW1lICgvZGV2L21kMikgaXMgdGhlbiB1 c2VkIGFzIGEgUFYgZm9yIExWTQ0KPiANCj4gSSBkbyBteSB3cml0ZSB0ZXN0cyBvbiBhIGZyZXNo IExWIHdpdGggbm8gZnJhZ21lbnRhdGlvbg0KPiANCj4gPiB0aW1lIHRoZSBORlMgY2xpZW50IHNl bmRzIGEgQ09NTUlUIG9yIHRydW5jKCkgaW5zdHJ1Y3Rpb24sIGFuZCBmb3INCj4gPiBwcmV0dHkg bXVjaCBhbGwgZmlsZSBhbmQgZGlyZWN0b3J5IGNyZWF0aW9uIG9wZXJhdGlvbnMgKHlvdSBjYW4g dXNlDQo+ID4gJ25mc3N0YXQnIHRvIG1vbml0b3IgaG93IG1hbnkgc3VjaCBvcGVyYXRpb25zIHRo ZSBORlMgY2xpZW50IGlzIHNlbmRpbmcNCj4gPiBhcyBwYXJ0IG9mIHlvdXIgdGVzdCkuDQo+IA0K PiBJIGtub3cgdGhhdCBteSB0d28gdGVzdHMgYXJlIHZlcnkgZGlmZmVyZW50IGluIHRoYXQgd2F5 Og0KPiANCj4gLSBkZCBpcyBqdXN0IHdyaXRpbmcgb25lIGJpZyBmaWxlLCBubyBmc3luYw0KPiAN Cj4gLSB1bnBhY2tpbmcgYSB0YXJiYWxsIChvciBjb21waWxpbmcgYSBsYXJnZSBDKysgcHJvamVj dCkgZG9lcyBhIGxvdCBvZg0KPiBzbWFsbCB3cml0ZXMgd2l0aCBtYW55IGZzeW5jcw0KPiANCj4g SW4gYm90aCBjYXNlcywgaXQgaXMgc2xvdw0KPiANCj4gPiBMb2NhbCBkaXNrIGNhbiBnZXQgYXdh eSB3aXRoIGRvaW5nIGEgbG90IGxlc3MgZnN5bmMoKSwgYmVjYXVzZSB0aGUgY2FjaGUNCj4gPiBj b25zaXN0ZW5jeSBndWFyYW50ZWVzIGFyZSBkaWZmZXJlbnQ6DQo+ID4gICAgICAgKiBpbiBORlMs IHRoZSBzZXJ2ZXIgaXMgYWxsb3dlZCB0byBjcmFzaCBvciByZWJvb3Qgd2l0aG91dA0KPiA+ICAg ICAgICAgYWZmZWN0aW5nIHRoZSBjbGllbnQncyB2aWV3IG9mIHRoZSBmaWxlc3lzdGVtLg0KPiA+ ICAgICAgICogaW4gdGhlIGxvY2FsIGZpbGUgc3lzdGVtLCB0aGUgZXhwZWN0YXRpb24gaXMgdGhh dCBvbiByZWJvb3QgYW55DQo+ID4gICAgICAgICBkYXRhIGxvc3QgaXMgd29uJ3QgbmVlZCB0byBi ZSByZWNvdmVyZWQgKHRoZSBhcHBsaWNhdGlvbiB3aWxsDQo+ID4gICAgICAgICBoYXZlIHVzZWQg ZnN5bmMoKSBmb3IgYW55IGRhdGEgdGhhdCBkb2VzIG5lZWQgdG8gYmUgcGVyc2lzdGVudCkuDQo+ ID4gICAgICAgICBPbmx5IHRoZSBkaXNrIGZpbGVzeXN0ZW0gc3RydWN0dXJlcyBuZWVkIHRvIGJl IHJlY292ZXJlZCwgYW5kDQo+ID4gICAgICAgICB0aGF0IGlzIGRvbmUgdXNpbmcgdGhlIGpvdXJu YWwgKG9yIGZzY2spLg0KPiANCj4gDQo+IElzIHRoaXMgYW4gaW50cmFjdGFibGUgcHJvYmxlbSB0 aG91Z2g/DQo+IA0KPiBPciBkbyBwZW9wbGUganVzdCB3b3JrIGFyb3VuZCB0aGlzLCBmb3IgZXhh bXBsZSwgZW5hYmxlIGFzeW5jIGFuZA0KPiB3cml0ZS1iYWNrIGNhY2hlLCBhbmQgdGhlbiB0cnkg dG8gbWFuYWdlIHRoZSByaXNrIGJ5IGFkZGluZyBhIFVQUyBhbmQvb3INCj4gYmF0dGVyeSBiYWNr ZWQgY2FjaGUgdG8gdGhlaXIgUkFJRCBzZXR1cCAodG8gcmVkdWNlIHRoZSBwcm9iYWJpbGl0eSBv Zg0KPiB1bmNsZWFuIHNodXRkb3duKT8NCg0KSXQgYWxsIGJvaWxzIGRvd24gdG8gd2hhdCBraW5k IG9mIGNvbnNpc3RlbmN5IGd1YXJhbnRlZXMgeW91IGFyZQ0KY29tZm9ydGFibGUgbGl2aW5nIHdp dGguIFRoZSBkZWZhdWx0IE5GUyBzZXJ2ZXIgc2V0dXAgb2ZmZXJzIG11Y2gNCnN0cm9uZ2VyIGRh dGEgY29uc2lzdGVuY3kgZ3VhcmFudGVlcyB0aGFuIGxvY2FsIGRpc2ssIGFuZCBpcyB0aGVyZWZv cmUNCmxpa2VseSB0byBiZSBzbG93ZXIgd2hlbiB1c2luZyBjaGVhcCBoYXJkd2FyZS4NCg0KLS0g DQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHAN ClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=
Quoted 24 lines · Trond
Quoted 23 lines · Daniel Pocock
Quoted 21 lines · Trond
Quoted 19 lines · Daniel Pocock
I've been observing some very slow nfs write performance when the server has `sync' in
/etc/exportsI want to avoid using async, but I have tested it and on my gigabit network, it gives almost the same speed as if I was on the server itself. (e.g. 30MB/sec to one disk, or less than 1MB/sec to the same disk over NFS with `sync')
I'm using Debian 6 with 2.6.38 kernels on client and server, NFSv3
I've also tried a client running Debian 7/Linux 3.2.0 with both NFSv3 and NFSv4, speed is still slow
Looking at iostat on the server, I notice that avgrq-sz = 8 sectors (4096 bytes) throughout the write operations
I've tried various tests, e.g. dd a large file, or unpack a tarball with many small files, the iostat output is always the same
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Were you using 'conv=sync'?
No, it was not using conv=sync, just the vanilla dd:
dd if=/dev/zero of=some-fat-file bs=65536 count=65536
Then the results are not comparable.
If I run dd with conv=sync on the server, then I still notice that OS caching plays a factor and write performance just appears really fast
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 11 lines
Looking at
/proc/mountson the clients, everything looks good, large wsize, tcp:rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.x.x.x,mountvers=3,mountport=58727,mountproto=udp,local_lock=none,addr=192.x.x.x 0 0
and
rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.x.x.x.,minorversion=0,local_lock=none,addr=192.x.x.x 0 0and in
/proc/fs/nfs/exportson the server, I have sync and wdelay:Quoted 7 lines
/nfs4/daniel 192.168.1.0/24,192.x.x.x(rw,insecure,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9,sec=1)
/home/daniel192.168.1.0/24,192.x.x.x(rw,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9)Can anyone suggest anything else? Or is this really the performance hit of `sync'?
It really depends on your disk setup. Particularly when your filesystem is using barriers (enabled by default on ext4 and xfs), a lot of raid
Quoted 8 lines
Quoted 5 lines
On the server, I've tried both ext3 and ext4, explicitly changing things like data=writeback,barrier=0, but the problem remains
The only thing that made it faster was using hdparm -W1 /dev/sd[ab] to enable the write-back cache on the disk
That should in principle be safe to do as long as you are using barrier=1.
Ok, so the combination of:
- enable writeback with hdparm
- use ext4 (and not ext3)
- barrier=1 and data=writeback? or data=?
- is there a particular kernel version (on either client or server side) that will offer more stability using this combination of features?
I think there are some other variations of my workflow that I can attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk because I don't need to keep the hundreds of object files.
Quoted 50 lines
Quoted 43 lines
Quoted 1 line
setups really _suck_ at dealing with
fsync(). The latter is used everyI'm using md RAID1, my setup is like this:
2x 1TB SATA disks ST31000528AS (7200rpm with 32MB cache and NCQ)
SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)
- not using any of the BIOS softraid stuffBoth devices have identical partitioning:
1. 128MB boot
2. md volume (1TB - 128MB)The entire md volume (
/dev/md2) is then used as a PV for LVMI do my write tests on a fresh LV with no fragmentation
Quoted 4 lines
time the NFS client sends a COMMIT or
trunc()instruction, and for pretty much all file and directory creation operations (you can use 'nfsstat' to monitor how many such operations the NFS client is sending as part of your test).I know that my two tests are very different in that way:
- dd is just writing one big file, no fsync
- unpacking a tarball (or compiling a large C++ project) does a lot of small writes with many fsyncs
In both cases, it is slow
Quoted 9 lines
Local disk can get away with doing a lot less
fsync(), because the cache consistency guarantees are different:
* in NFS, the server is allowed to crash or reboot without affecting the client's view of the filesystem.
* in the local file system, the expectation is that on reboot any data lost is won't need to be recovered (the application will have usedfsync()for any data that does need to be persistent). Only the disk filesystem structures need to be recovered, and that is done using the journal (or fsck).Quoted 6 lines
Is this an intractable problem though?
Or do people just work around this, for example, enable async and write-back cache, and then try to manage the risk by adding a UPS and/or battery backed cache to their RAID setup (to reduce the probability of unclean shutdown)?
Quoted 5 lines
It all boils down to what kind of consistency guarantees you are comfortable living with. The default NFS server setup offers much stronger data consistency guarantees than local disk, and is therefore likely to be slower when using cheap hardware.
I'm keen for consistency, because I don't like the idea of corrupting some source code or a whole git repository for example.
How did you know I'm using cheap hardware? It is a HP MicroServer, I even got the £100 cash-back cheque:
http://www8.hp.com/uk/en/campaign/focus-for-smb/solution.html#/tab2/
Seriously though, I've worked with some very large arrays in my business environment, but I use this hardware at home because of the low noise and low heat dissipation rather than for saving money, so I would like to try and get the most out of it if possible.
Quoted 24 lines · Trond
Quoted 23 lines · Daniel Pocock
Quoted 21 lines · Trond
Quoted 19 lines · Daniel Pocock
I've been observing some very slow nfs write performance when the server has `sync' in
/etc/exportsI want to avoid using async, but I have tested it and on my gigabit network, it gives almost the same speed as if I was on the server itself. (e.g. 30MB/sec to one disk, or less than 1MB/sec to the same disk over NFS with `sync')
I'm using Debian 6 with 2.6.38 kernels on client and server, NFSv3
I've also tried a client running Debian 7/Linux 3.2.0 with both NFSv3 and NFSv4, speed is still slow
Looking at iostat on the server, I notice that avgrq-sz = 8 sectors (4096 bytes) throughout the write operations
I've tried various tests, e.g. dd a large file, or unpack a tarball with many small files, the iostat output is always the same
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Were you using 'conv=sync'?
No, it was not using conv=sync, just the vanilla dd:
dd if=/dev/zero of=some-fat-file bs=65536 count=65536
Then the results are not comparable.
If I run dd with conv=sync on the server, then I still notice that OS caching plays a factor and write performance just appears really fast
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 11 lines
Looking at
/proc/mountson the clients, everything looks good, large wsize, tcp:rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.x.x.x,mountvers=3,mountport=58727,mountproto=udp,local_lock=none,addr=192.x.x.x 0 0
and
rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.x.x.x.,minorversion=0,local_lock=none,addr=192.x.x.x 0 0and in
/proc/fs/nfs/exportson the server, I have sync and wdelay:Quoted 7 lines
/nfs4/daniel 192.168.1.0/24,192.x.x.x(rw,insecure,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9,sec=1)
/home/daniel192.168.1.0/24,192.x.x.x(rw,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9)Can anyone suggest anything else? Or is this really the performance hit of `sync'?
It really depends on your disk setup. Particularly when your filesystem is using barriers (enabled by default on ext4 and xfs), a lot of raid
Quoted 8 lines
Quoted 5 lines
On the server, I've tried both ext3 and ext4, explicitly changing things like data=writeback,barrier=0, but the problem remains
The only thing that made it faster was using hdparm -W1 /dev/sd[ab] to enable the write-back cache on the disk
That should in principle be safe to do as long as you are using barrier=1.
Ok, so the combination of:
- enable writeback with hdparm
- use ext4 (and not ext3)
- barrier=1 and data=writeback? or data=?
- is there a particular kernel version (on either client or server side) that will offer more stability using this combination of features?
I think there are some other variations of my workflow that I can attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk because I don't need to keep the hundreds of object files.
Quoted 50 lines
Quoted 43 lines
Quoted 1 line
setups really _suck_ at dealing with
fsync(). The latter is used everyI'm using md RAID1, my setup is like this:
2x 1TB SATA disks ST31000528AS (7200rpm with 32MB cache and NCQ)
SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)
- not using any of the BIOS softraid stuffBoth devices have identical partitioning:
1. 128MB boot
2. md volume (1TB - 128MB)The entire md volume (
/dev/md2) is then used as a PV for LVMI do my write tests on a fresh LV with no fragmentation
Quoted 4 lines
time the NFS client sends a COMMIT or
trunc()instruction, and for pretty much all file and directory creation operations (you can use 'nfsstat' to monitor how many such operations the NFS client is sending as part of your test).I know that my two tests are very different in that way:
- dd is just writing one big file, no fsync
- unpacking a tarball (or compiling a large C++ project) does a lot of small writes with many fsyncs
In both cases, it is slow
Quoted 9 lines
Local disk can get away with doing a lot less
fsync(), because the cache consistency guarantees are different:
* in NFS, the server is allowed to crash or reboot without affecting the client's view of the filesystem.
* in the local file system, the expectation is that on reboot any data lost is won't need to be recovered (the application will have usedfsync()for any data that does need to be persistent). Only the disk filesystem structures need to be recovered, and that is done using the journal (or fsck).Quoted 6 lines
Is this an intractable problem though?
Or do people just work around this, for example, enable async and write-back cache, and then try to manage the risk by adding a UPS and/or battery backed cache to their RAID setup (to reduce the probability of unclean shutdown)?
Quoted 5 lines
It all boils down to what kind of consistency guarantees you are comfortable living with. The default NFS server setup offers much stronger data consistency guarantees than local disk, and is therefore likely to be slower when using cheap hardware.
I'm keen for consistency, because I don't like the idea of corrupting some source code or a whole git repository for example.
How did you know I'm using cheap hardware? It is a HP MicroServer, I even got the £100 cash-back cheque:
http://www8.hp.com/uk/en/campaign/focus-for-smb/solution.html#/tab2/
Seriously though, I've worked with some very large arrays in my business environment, but I use this hardware at home because of the low noise and low heat dissipation rather than for saving money, so I would like to try and get the most out of it if possible and I'm very grateful for these suggestions.
T24gU3VuLCAyMDEyLTA1LTA2IGF0IDIyOjEyICswMDAwLCBEYW5pZWwgUG9jb2NrIHdyb3RlOg0K PiANCj4gT24gMDYvMDUvMTIgMjE6NDksIE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+ID4gT24g U3VuLCAyMDEyLTA1LTA2IGF0IDIxOjIzICswMDAwLCBEYW5pZWwgUG9jb2NrIHdyb3RlOg0KPiA+ Pg0KPiA+PiBPbiAwNi8wNS8xMiAxODoyMywgTXlrbGVidXN0LCBUcm9uZCB3cm90ZToNCj4gPj4+ IE9uIFN1biwgMjAxMi0wNS0wNiBhdCAwMzowMCArMDAwMCwgRGFuaWVsIFBvY29jayB3cm90ZToN Cj4gPj4+Pg0KPiA+Pj4+IEkndmUgYmVlbiBvYnNlcnZpbmcgc29tZSB2ZXJ5IHNsb3cgbmZzIHdy aXRlIHBlcmZvcm1hbmNlIHdoZW4gdGhlIHNlcnZlcg0KPiA+Pj4+IGhhcyBgc3luYycgaW4gL2V0 Yy9leHBvcnRzDQo+ID4+Pj4NCj4gPj4+PiBJIHdhbnQgdG8gYXZvaWQgdXNpbmcgYXN5bmMsIGJ1 dCBJIGhhdmUgdGVzdGVkIGl0IGFuZCBvbiBteSBnaWdhYml0DQo+ID4+Pj4gbmV0d29yaywgaXQg Z2l2ZXMgYWxtb3N0IHRoZSBzYW1lIHNwZWVkIGFzIGlmIEkgd2FzIG9uIHRoZSBzZXJ2ZXINCj4g Pj4+PiBpdHNlbGYuIChlLmcuIDMwTUIvc2VjIHRvIG9uZSBkaXNrLCBvciBsZXNzIHRoYW4gMU1C L3NlYyB0byB0aGUgc2FtZQ0KPiA+Pj4+IGRpc2sgb3ZlciBORlMgd2l0aCBgc3luYycpDQo+ID4+ Pj4NCj4gPj4+PiBJJ20gdXNpbmcgRGViaWFuIDYgd2l0aCAyLjYuMzgga2VybmVscyBvbiBjbGll bnQgYW5kIHNlcnZlciwgTkZTdjMNCj4gPj4+Pg0KPiA+Pj4+IEkndmUgYWxzbyB0cmllZCBhIGNs aWVudCBydW5uaW5nIERlYmlhbiA3L0xpbnV4IDMuMi4wIHdpdGggYm90aCBORlN2Mw0KPiA+Pj4+ IGFuZCBORlN2NCwgc3BlZWQgaXMgc3RpbGwgc2xvdw0KPiA+Pj4+DQo+ID4+Pj4gTG9va2luZyBh dCBpb3N0YXQgb24gdGhlIHNlcnZlciwgSSBub3RpY2UgdGhhdCBhdmdycS1zeiA9IDggc2VjdG9y cw0KPiA+Pj4+ICg0MDk2IGJ5dGVzKSB0aHJvdWdob3V0IHRoZSB3cml0ZSBvcGVyYXRpb25zDQo+ ID4+Pj4NCj4gPj4+PiBJJ3ZlIHRyaWVkIHZhcmlvdXMgdGVzdHMsIGUuZy4gZGQgYSBsYXJnZSBm aWxlLCBvciB1bnBhY2sgYSB0YXJiYWxsIHdpdGgNCj4gPj4+PiBtYW55IHNtYWxsIGZpbGVzLCB0 aGUgaW9zdGF0IG91dHB1dCBpcyBhbHdheXMgdGhlIHNhbWUNCj4gPj4+DQo+ID4+PiBXZXJlIHlv dSB1c2luZyAnY29udj1zeW5jJz8NCj4gPj4NCj4gPj4gTm8sIGl0IHdhcyBub3QgdXNpbmcgY29u dj1zeW5jLCBqdXN0IHRoZSB2YW5pbGxhIGRkOg0KPiA+Pg0KPiA+PiBkZCBpZj0vZGV2L3plcm8g b2Y9c29tZS1mYXQtZmlsZSBicz02NTUzNiBjb3VudD02NTUzNg0KPiA+IA0KPiA+IFRoZW4gdGhl IHJlc3VsdHMgYXJlIG5vdCBjb21wYXJhYmxlLg0KPiANCj4gSWYgSSBydW4gZGQgd2l0aCBjb252 PXN5bmMgb24gdGhlIHNlcnZlciwgdGhlbiBJIHN0aWxsIG5vdGljZSB0aGF0IE9TDQo+IGNhY2hp bmcgcGxheXMgYSBmYWN0b3IgYW5kIHdyaXRlIHBlcmZvcm1hbmNlIGp1c3QgYXBwZWFycyByZWFs bHkgZmFzdA0KPiANCj4gPj4+PiBMb29raW5nIGF0IC9wcm9jL21vdW50cyBvbiB0aGUgY2xpZW50 cywgZXZlcnl0aGluZyBsb29rcyBnb29kLCBsYXJnZQ0KPiA+Pj4+IHdzaXplLCB0Y3A6DQo+ID4+ Pj4NCj4gPj4+PiBydyxyZWxhdGltZSx2ZXJzPTMscnNpemU9MTA0ODU3Nix3c2l6ZT0xMDQ4NTc2 LG5hbWxlbj0yNTUsaGFyZCxwcm90bz10Y3AsdGltZW89NjAwLHJldHJhbnM9MixzZWM9c3lzLG1v dW50YWRkcj0xOTIueC54LngsbW91bnR2ZXJzPTMsbW91bnRwb3J0PTU4NzI3LG1vdW50cHJvdG89 dWRwLGxvY2FsX2xvY2s9bm9uZSxhZGRyPTE5Mi54LngueA0KPiA+Pj4+IDAgMA0KPiA+Pj4+DQo+ ID4+Pj4gYW5kDQo+ID4+Pj4gIHJ3LHJlbGF0aW1lLHZlcnM9NCxyc2l6ZT0xMDQ4NTc2LHdzaXpl PTEwNDg1NzYsbmFtbGVuPTI1NSxoYXJkLHByb3RvPXRjcCxwb3J0PTAsdGltZW89NjAwLHJldHJh bnM9MixzZWM9c3lzLGNsaWVudGFkZHI9MTkyLngueC54LixtaW5vcnZlcnNpb249MCxsb2NhbF9s b2NrPW5vbmUsYWRkcj0xOTIueC54LnggMCAwDQo+ID4+Pj4NCj4gPj4+PiBhbmQgaW4gL3Byb2Mv ZnMvbmZzL2V4cG9ydHMgb24gdGhlIHNlcnZlciwgSSBoYXZlIHN5bmMgYW5kIHdkZWxheToNCj4g Pj4+Pg0KPiA+Pj4+IC9uZnM0L2RhbmllbA0KPiA+Pj4+IDE5Mi4xNjguMS4wLzI0LDE5Mi54Lngu eChydyxpbnNlY3VyZSxyb290X3NxdWFzaCxzeW5jLHdkZWxheSxub19zdWJ0cmVlX2NoZWNrLHV1 aWQ9YWEyYTZmMzc6OWNjOTRlZWI6YmNiZjk4M2M6ZDZlMDQxZDksc2VjPTEpDQo+ID4+Pj4gL2hv bWUvZGFuaWVsDQo+ID4+Pj4gMTkyLjE2OC4xLjAvMjQsMTkyLngueC54KHJ3LHJvb3Rfc3F1YXNo LHN5bmMsd2RlbGF5LG5vX3N1YnRyZWVfY2hlY2ssdXVpZD1hYTJhNmYzNzo5Y2M5NGVlYjpiY2Jm OTgzYzpkNmUwNDFkOSkNCj4gPj4+Pg0KPiA+Pj4+IENhbiBhbnlvbmUgc3VnZ2VzdCBhbnl0aGlu ZyBlbHNlPyAgT3IgaXMgdGhpcyByZWFsbHkgdGhlIHBlcmZvcm1hbmNlIGhpdA0KPiA+Pj4+IG9m IGBzeW5jJz8NCj4gPj4+DQo+ID4+PiBJdCByZWFsbHkgZGVwZW5kcyBvbiB5b3VyIGRpc2sgc2V0 dXAuIFBhcnRpY3VsYXJseSB3aGVuIHlvdXIgZmlsZXN5c3RlbQ0KPiA+Pj4gaXMgdXNpbmcgYmFy cmllcnMgKGVuYWJsZWQgYnkgZGVmYXVsdCBvbiBleHQ0IGFuZCB4ZnMpLCBhIGxvdCBvZiByYWlk DQo+ID4+DQo+ID4+IE9uIHRoZSBzZXJ2ZXIsIEkndmUgdHJpZWQgYm90aCBleHQzIGFuZCBleHQ0 LCBleHBsaWNpdGx5IGNoYW5naW5nIHRoaW5ncw0KPiA+PiBsaWtlIGRhdGE9d3JpdGViYWNrLGJh cnJpZXI9MCwgYnV0IHRoZSBwcm9ibGVtIHJlbWFpbnMNCj4gPj4NCj4gPj4gVGhlIG9ubHkgdGhp bmcgdGhhdCBtYWRlIGl0IGZhc3RlciB3YXMgdXNpbmcgaGRwYXJtIC1XMSAvZGV2L3NkW2FiXSB0 bw0KPiA+PiBlbmFibGUgdGhlIHdyaXRlLWJhY2sgY2FjaGUgb24gdGhlIGRpc2sNCj4gPiANCj4g PiBUaGF0IHNob3VsZCBpbiBwcmluY2lwbGUgYmUgc2FmZSB0byBkbyBhcyBsb25nIGFzIHlvdSBh cmUgdXNpbmcNCj4gPiBiYXJyaWVyPTEuDQo+IA0KPiBPaywgc28gdGhlIGNvbWJpbmF0aW9uIG9m Og0KPiANCj4gLSBlbmFibGUgd3JpdGViYWNrIHdpdGggaGRwYXJtDQo+IC0gdXNlIGV4dDQgKGFu ZCBub3QgZXh0MykNCj4gLSBiYXJyaWVyPTEgYW5kIGRhdGE9d3JpdGViYWNrPyAgb3IgZGF0YT0/ DQo+IA0KPiAtIGlzIHRoZXJlIGEgcGFydGljdWxhciBrZXJuZWwgdmVyc2lvbiAob24gZWl0aGVy IGNsaWVudCBvciBzZXJ2ZXIgc2lkZSkNCj4gdGhhdCB3aWxsIG9mZmVyIG1vcmUgc3RhYmlsaXR5 IHVzaW5nIHRoaXMgY29tYmluYXRpb24gb2YgZmVhdHVyZXM/DQoNCk5vdCB0aGF0IEknbSBhd2Fy ZSBvZi4gQXMgbG9uZyBhcyB5b3UgaGF2ZSBhIGtlcm5lbCA+IDIuNi4yOSwgdGhlbiBMVk0NCnNo b3VsZCB3b3JrIGNvcnJlY3RseS4gVGhlIG1haW4gcHJvYmxlbSBpcyB0aGF0IHNvbWUgU0FUQSBo YXJkd2FyZSB0ZW5kcw0KdG8gYmUgYnVnZ3ksIGRlZmVhdGluZyB0aGUgbWV0aG9kcyB1c2VkIGJ5 IHRoZSBiYXJyaWVyIGNvZGUgdG8gZW5zdXJlDQpkYXRhIGlzIHRydWx5IG9uIGRpc2suIEkgYmVs aWV2ZSB0aGF0IFhGUyB3aWxsIHRoZXJlZm9yZSBhY3R1YWxseSB0ZXN0DQp0aGUgaGFyZHdhcmUg d2hlbiB5b3UgbW91bnQgd2l0aCB3cml0ZSBjYWNoaW5nIGFuZCBiYXJyaWVycywgYW5kIHNob3Vs ZA0KcmVwb3J0IGlmIHRoZSB0ZXN0IGZhaWxzIGluIHRoZSBzeXNsb2dzLg0KU2VlIGh0dHA6Ly94 ZnMub3JnL2luZGV4LnBocC9YRlNfRkFRI1dyaXRlX2JhcnJpZXJfc3VwcG9ydC4NCg0KPiBJIHRo aW5rIHRoZXJlIGFyZSBzb21lIG90aGVyIHZhcmlhdGlvbnMgb2YgbXkgd29ya2Zsb3cgdGhhdCBJ IGNhbg0KPiBhdHRlbXB0IHRvbywgZS5nLiBJJ3ZlIGNvbnRlbXBsYXRlZCBjb21waWxpbmcgQysr IGNvZGUgb250byBhIFJBTSBkaXNrDQo+IGJlY2F1c2UgSSBkb24ndCBuZWVkIHRvIGtlZXAgdGhl IGh1bmRyZWRzIG9mIG9iamVjdCBmaWxlcy4NCg0KWW91IG1pZ2h0IGFsc28gY29uc2lkZXIgdXNp bmcgc29tZXRoaW5nIGxpa2UgY2NhY2hlIGFuZCBzZXQgdGhlDQpDQ0FDSEVfRElSIHRvIGEgbG9j YWwgZGlzayBpZiB5b3UgaGF2ZSBvbmUuDQoNCj4gPj4+IHNldHVwcyByZWFsbHkgX3N1Y2tfIGF0 IGRlYWxpbmcgd2l0aCBmc3luYygpLiBUaGUgbGF0dGVyIGlzIHVzZWQgZXZlcnkNCj4gPj4NCj4g Pj4gSSdtIHVzaW5nIG1kIFJBSUQxLCBteSBzZXR1cCBpcyBsaWtlIHRoaXM6DQo+ID4+DQo+ID4+ IDJ4IDFUQiBTQVRBIGRpc2tzIFNUMzEwMDA1MjhBUyAoNzIwMHJwbSB3aXRoIDMyTUIgY2FjaGUg YW5kIE5DUSkNCj4gPj4NCj4gPj4gU0FUQSBjb250cm9sbGVyOiBBVEkgVGVjaG5vbG9naWVzIElu YyBTQjcwMC9TQjgwMCBTQVRBIENvbnRyb2xsZXIgW0FIQ0kNCj4gPj4gbW9kZV0gKHJldiA0MCkN Cj4gPj4gLSBub3QgdXNpbmcgYW55IG9mIHRoZSBCSU9TIHNvZnRyYWlkIHN0dWZmDQo+ID4+DQo+ ID4+IEJvdGggZGV2aWNlcyBoYXZlIGlkZW50aWNhbCBwYXJ0aXRpb25pbmc6DQo+ID4+IDEuIDEy OE1CIGJvb3QNCj4gPj4gMi4gbWQgdm9sdW1lICgxVEIgLSAxMjhNQikNCj4gPj4NCj4gPj4gVGhl IGVudGlyZSBtZCB2b2x1bWUgKC9kZXYvbWQyKSBpcyB0aGVuIHVzZWQgYXMgYSBQViBmb3IgTFZN DQo+ID4+DQo+ID4+IEkgZG8gbXkgd3JpdGUgdGVzdHMgb24gYSBmcmVzaCBMViB3aXRoIG5vIGZy YWdtZW50YXRpb24NCj4gPj4NCj4gPj4+IHRpbWUgdGhlIE5GUyBjbGllbnQgc2VuZHMgYSBDT01N SVQgb3IgdHJ1bmMoKSBpbnN0cnVjdGlvbiwgYW5kIGZvcg0KPiA+Pj4gcHJldHR5IG11Y2ggYWxs IGZpbGUgYW5kIGRpcmVjdG9yeSBjcmVhdGlvbiBvcGVyYXRpb25zICh5b3UgY2FuIHVzZQ0KPiA+ Pj4gJ25mc3N0YXQnIHRvIG1vbml0b3IgaG93IG1hbnkgc3VjaCBvcGVyYXRpb25zIHRoZSBORlMg Y2xpZW50IGlzIHNlbmRpbmcNCj4gPj4+IGFzIHBhcnQgb2YgeW91ciB0ZXN0KS4NCj4gPj4NCj4g Pj4gSSBrbm93IHRoYXQgbXkgdHdvIHRlc3RzIGFyZSB2ZXJ5IGRpZmZlcmVudCBpbiB0aGF0IHdh eToNCj4gPj4NCj4gPj4gLSBkZCBpcyBqdXN0IHdyaXRpbmcgb25lIGJpZyBmaWxlLCBubyBmc3lu Yw0KPiA+Pg0KPiA+PiAtIHVucGFja2luZyBhIHRhcmJhbGwgKG9yIGNvbXBpbGluZyBhIGxhcmdl IEMrKyBwcm9qZWN0KSBkb2VzIGEgbG90IG9mDQo+ID4+IHNtYWxsIHdyaXRlcyB3aXRoIG1hbnkg ZnN5bmNzDQo+ID4+DQo+ID4+IEluIGJvdGggY2FzZXMsIGl0IGlzIHNsb3cNCj4gPj4NCj4gPj4+ IExvY2FsIGRpc2sgY2FuIGdldCBhd2F5IHdpdGggZG9pbmcgYSBsb3QgbGVzcyBmc3luYygpLCBi ZWNhdXNlIHRoZSBjYWNoZQ0KPiA+Pj4gY29uc2lzdGVuY3kgZ3VhcmFudGVlcyBhcmUgZGlmZmVy ZW50Og0KPiA+Pj4gICAgICAgKiBpbiBORlMsIHRoZSBzZXJ2ZXIgaXMgYWxsb3dlZCB0byBjcmFz aCBvciByZWJvb3Qgd2l0aG91dA0KPiA+Pj4gICAgICAgICBhZmZlY3RpbmcgdGhlIGNsaWVudCdz IHZpZXcgb2YgdGhlIGZpbGVzeXN0ZW0uDQo+ID4+PiAgICAgICAqIGluIHRoZSBsb2NhbCBmaWxl IHN5c3RlbSwgdGhlIGV4cGVjdGF0aW9uIGlzIHRoYXQgb24gcmVib290IGFueQ0KPiA+Pj4gICAg ICAgICBkYXRhIGxvc3QgaXMgd29uJ3QgbmVlZCB0byBiZSByZWNvdmVyZWQgKHRoZSBhcHBsaWNh dGlvbiB3aWxsDQo+ID4+PiAgICAgICAgIGhhdmUgdXNlZCBmc3luYygpIGZvciBhbnkgZGF0YSB0 aGF0IGRvZXMgbmVlZCB0byBiZSBwZXJzaXN0ZW50KS4NCj4gPj4+ICAgICAgICAgT25seSB0aGUg ZGlzayBmaWxlc3lzdGVtIHN0cnVjdHVyZXMgbmVlZCB0byBiZSByZWNvdmVyZWQsIGFuZA0KPiA+ Pj4gICAgICAgICB0aGF0IGlzIGRvbmUgdXNpbmcgdGhlIGpvdXJuYWwgKG9yIGZzY2spLg0KPiA+ Pg0KPiA+Pg0KPiA+PiBJcyB0aGlzIGFuIGludHJhY3RhYmxlIHByb2JsZW0gdGhvdWdoPw0KPiA+ Pg0KPiA+PiBPciBkbyBwZW9wbGUganVzdCB3b3JrIGFyb3VuZCB0aGlzLCBmb3IgZXhhbXBsZSwg ZW5hYmxlIGFzeW5jIGFuZA0KPiA+PiB3cml0ZS1iYWNrIGNhY2hlLCBhbmQgdGhlbiB0cnkgdG8g bWFuYWdlIHRoZSByaXNrIGJ5IGFkZGluZyBhIFVQUyBhbmQvb3INCj4gPj4gYmF0dGVyeSBiYWNr ZWQgY2FjaGUgdG8gdGhlaXIgUkFJRCBzZXR1cCAodG8gcmVkdWNlIHRoZSBwcm9iYWJpbGl0eSBv Zg0KPiA+PiB1bmNsZWFuIHNodXRkb3duKT8NCj4gPiANCj4gPiBJdCBhbGwgYm9pbHMgZG93biB0 byB3aGF0IGtpbmQgb2YgY29uc2lzdGVuY3kgZ3VhcmFudGVlcyB5b3UgYXJlDQo+ID4gY29tZm9y dGFibGUgbGl2aW5nIHdpdGguIFRoZSBkZWZhdWx0IE5GUyBzZXJ2ZXIgc2V0dXAgb2ZmZXJzIG11 Y2gNCj4gPiBzdHJvbmdlciBkYXRhIGNvbnNpc3RlbmN5IGd1YXJhbnRlZXMgdGhhbiBsb2NhbCBk aXNrLCBhbmQgaXMgdGhlcmVmb3JlDQo+ID4gbGlrZWx5IHRvIGJlIHNsb3dlciB3aGVuIHVzaW5n IGNoZWFwIGhhcmR3YXJlLg0KPiA+IA0KPiANCj4gSSdtIGtlZW4gZm9yIGNvbnNpc3RlbmN5LCBi ZWNhdXNlIEkgZG9uJ3QgbGlrZSB0aGUgaWRlYSBvZiBjb3JydXB0aW5nDQo+IHNvbWUgc291cmNl IGNvZGUgb3IgYSB3aG9sZSBnaXQgcmVwb3NpdG9yeSBmb3IgZXhhbXBsZS4NCj4gDQo+IEhvdyBk aWQgeW91IGtub3cgSSdtIHVzaW5nIGNoZWFwIGhhcmR3YXJlPyAgSXQgaXMgYSBIUCBNaWNyb1Nl cnZlciwgSQ0KPiBldmVuIGdvdCB0aGUgwqMxMDAgY2FzaC1iYWNrIGNoZXF1ZToNCj4gDQo+IGh0 dHA6Ly93d3c4LmhwLmNvbS91ay9lbi9jYW1wYWlnbi9mb2N1cy1mb3Itc21iL3NvbHV0aW9uLmh0 bWwjL3RhYjIvDQo+IA0KPiBTZXJpb3VzbHkgdGhvdWdoLCBJJ3ZlIHdvcmtlZCB3aXRoIHNvbWUg dmVyeSBsYXJnZSBhcnJheXMgaW4gbXkgYnVzaW5lc3MNCj4gZW52aXJvbm1lbnQsIGJ1dCBJIHVz ZSB0aGlzIGhhcmR3YXJlIGF0IGhvbWUgYmVjYXVzZSBvZiB0aGUgbG93IG5vaXNlDQo+IGFuZCBs b3cgaGVhdCBkaXNzaXBhdGlvbiByYXRoZXIgdGhhbiBmb3Igc2F2aW5nIG1vbmV5LCBzbyBJIHdv dWxkIGxpa2UNCj4gdG8gdHJ5IGFuZCBnZXQgdGhlIG1vc3Qgb3V0IG9mIGl0IGlmIHBvc3NpYmxl IGFuZCBJJ20gdmVyeSBncmF0ZWZ1bCBmb3INCj4gdGhlc2Ugc3VnZ2VzdGlvbnMuDQoNClJpZ2h0 LiBBbGwgSSdtIHNheWluZyBpcyB0aGF0IHdoZW4gY29tcGFyaW5nIGxvY2FsIGRpc2sgYW5kIE5G Uw0KcGVyZm9ybWFuY2UsIHRoZW4gbWFrZSBzdXJlIHRoYXQgeW91IGFyZSBkb2luZyBhbiBhcHBs ZXMtdG8tYXBwbGVzDQpjb21wYXJpc29uLg0KVGhlIG1haW4gcmVhc29uIGZvciB3YW50aW5nIHRv IHVzZSBORlMgaW4gYSBob21lIHNldHVwIHdvdWxkIHVzdWFsbHkgYmUNCmluIG9yZGVyIHRvIHNp bXVsdGFuZW91c2x5IGFjY2VzcyB0aGUgc2FtZSBkYXRhIHRocm91Z2ggc2V2ZXJhbCBjbGllbnRz Lg0KSWYgdGhhdCBpcyBub3QgYSBjb25jZXJuLCB0aGVuIHBlcmhhcHMgdHJhbnNmb3JtaW5nIHlv dXIgTkZTIHNlcnZlciBpbnRvDQphbiBpU0NTSSB0YXJnZXQgbWlnaHQgZml0IHlvdXIgcGVyZm9y bWFuY2UgcmVxdWlyZW1lbnRzIGJldHRlcj8NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4 IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAu Y29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=
Quoted 18 lines
Quoted 9 lines
Ok, so the combination of:
- enable writeback with hdparm
- use ext4 (and not ext3)
- barrier=1 and data=writeback? or data=?- is there a particular kernel version (on either client or server side) that will offer more stability using this combination of features?
Not that I'm aware of. As long as you have a kernel > 2.6.29, then LVM should work correctly. The main problem is that some SATA hardware tends to be buggy, defeating the methods used by the barrier code to ensure data is truly on disk. I believe that XFS will therefore actually test the hardware when you mount with write caching and barriers, and should report if the test fails in the syslogs.
See http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.
Quoted 7 lines
Quoted 3 lines
I think there are some other variations of my workflow that I can attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk because I don't need to keep the hundreds of object files.
You might also consider using something like ccache and set the
CCACHE_DIRto a local disk if you have one.
Thanks for the feedback about these options, I am going to look at these strategies more closely
Quoted 70 lines
Quoted 62 lines
Quoted 50 lines
Quoted 43 lines
Quoted 1 line
setups really _suck_ at dealing with
fsync(). The latter is used everyI'm using md RAID1, my setup is like this:
2x 1TB SATA disks ST31000528AS (7200rpm with 32MB cache and NCQ)
SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)
- not using any of the BIOS softraid stuffBoth devices have identical partitioning:
1. 128MB boot
2. md volume (1TB - 128MB)The entire md volume (
/dev/md2) is then used as a PV for LVMI do my write tests on a fresh LV with no fragmentation
Quoted 4 lines
time the NFS client sends a COMMIT or
trunc()instruction, and for pretty much all file and directory creation operations (you can use 'nfsstat' to monitor how many such operations the NFS client is sending as part of your test).I know that my two tests are very different in that way:
- dd is just writing one big file, no fsync
- unpacking a tarball (or compiling a large C++ project) does a lot of small writes with many fsyncs
In both cases, it is slow
Quoted 9 lines
Local disk can get away with doing a lot less
fsync(), because the cache consistency guarantees are different:
* in NFS, the server is allowed to crash or reboot without affecting the client's view of the filesystem.
* in the local file system, the expectation is that on reboot any data lost is won't need to be recovered (the application will have usedfsync()for any data that does need to be persistent). Only the disk filesystem structures need to be recovered, and that is done using the journal (or fsck).Quoted 6 lines
Is this an intractable problem though?
Or do people just work around this, for example, enable async and write-back cache, and then try to manage the risk by adding a UPS and/or battery backed cache to their RAID setup (to reduce the probability of unclean shutdown)?
Quoted 5 lines
It all boils down to what kind of consistency guarantees you are comfortable living with. The default NFS server setup offers much stronger data consistency guarantees than local disk, and is therefore likely to be slower when using cheap hardware.
I'm keen for consistency, because I don't like the idea of corrupting some source code or a whole git repository for example.
How did you know I'm using cheap hardware? It is a HP MicroServer, I even got the £100 cash-back cheque:
Quoted 7 lines
http://www8.hp.com/uk/en/campaign/focus-for-smb/solution.html#/tab2/
Seriously though, I've worked with some very large arrays in my business environment, but I use this hardware at home because of the low noise and low heat dissipation rather than for saving money, so I would like to try and get the most out of it if possible and I'm very grateful for these suggestions.
Quoted 7 lines
Right. All I'm saying is that when comparing local disk and NFS performance, then make sure that you are doing an apples-to-apples comparison.
The main reason for wanting to use NFS in a home setup would usually be in order to simultaneously access the same data through several clients. If that is not a concern, then perhaps transforming your NFS server into an iSCSI target might fit your performance requirements better?
There are various types of content, some things, like VM images, are only accessed by one client at a time, those could be iSCSI.
However, some of the code I'm compiling needs to be built on different platforms (e.g. I have a Debian squeeze desktop, and Debian wheezy in a VM), consequently, it is convenient for me to have access to the git workspaces from all these hosts using NFS.
Quoted 22 lines · Daniel Pocock
Quoted 22 lines
Quoted 9 lines
Ok, so the combination of:
- enable writeback with hdparm
- use ext4 (and not ext3)
- barrier=1 and data=writeback? or data=?- is there a particular kernel version (on either client or server side) that will offer more stability using this combination of features?
Not that I'm aware of. As long as you have a kernel > 2.6.29, then LVM should work correctly. The main problem is that some SATA hardware tends to be buggy, defeating the methods used by the barrier code to ensure data is truly on disk. I believe that XFS will therefore actually test the hardware when you mount with write caching and barriers, and should report if the test fails in the syslogs.
See http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.Quoted 3 lines
I think there are some other variations of my workflow that I can attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk because I don't need to keep the hundreds of object files.
Quoted 8 lines
Quoted 3 lines
You might also consider using something like ccache and set the
CCACHE_DIRto a local disk if you have one.Thanks for the feedback about these options, I am going to look at these strategies more closely
I decided to try and take md and LVM out of the picture, I tried two variations:
a) the boot partitions are not mirrored, so I reformatted one of them as ext4,
- enabled write-cache for the whole of sdb,
- mounted ext4, barrier=1,data=ordered
- and exported this volume over NFS
unpacking a large source tarball on this volume, iostat reports write speeds that are even slower, barely 300kBytes/sec
b) I took an external USB HDD,
- created two 20GB partitions sdc1 and sdc2
- formatted sdc1 as btrfs
- formatted sdc2 as ext4
- mounted sdc2 the same as sdb1 in test (a),
ext4, barrier=1,data=ordered
- exported both volumes over NFS
unpacking a large source tarball on these two volumes, iostat reports write speeds that are around 5MB/sec - much faster than the original problem I was having
Bottom line, this leaves me with the impression that either
- the server's SATA controller or disks need a firmware upgrade,
- or there is some issue with the kernel barriers and/or cache flushing on this specific SATA hardware.
I think it is fair to say that the NFS client is not at fault, however, I can imagine many people would be tempted to just use `async' when faced with a problem like this, given that async makes everything just run fast.
Quoted 38 lines · Daniel Pocock
Quoted 22 lines · Daniel Pocock
Quoted 22 lines
Quoted 9 lines
Ok, so the combination of:
- enable writeback with hdparm
- use ext4 (and not ext3)
- barrier=1 and data=writeback? or data=?- is there a particular kernel version (on either client or server side) that will offer more stability using this combination of features?
Not that I'm aware of. As long as you have a kernel > 2.6.29, then LVM should work correctly. The main problem is that some SATA hardware tends to be buggy, defeating the methods used by the barrier code to ensure data is truly on disk. I believe that XFS will therefore actually test the hardware when you mount with write caching and barriers, and should report if the test fails in the syslogs.
See http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.Quoted 3 lines
I think there are some other variations of my workflow that I can attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk because I don't need to keep the hundreds of object files.
Quoted 8 lines
Quoted 3 lines
You might also consider using something like ccache and set the
CCACHE_DIRto a local disk if you have one.Thanks for the feedback about these options, I am going to look at these strategies more closely
I decided to try and take md and LVM out of the picture, I tried two variations:
Quoted 8 lines
a) the boot partitions are not mirrored, so I reformatted one of them as ext4,
- enabled write-cache for the whole of sdb,
- mounted ext4, barrier=1,data=ordered
- and exported this volume over NFSunpacking a large source tarball on this volume, iostat reports write speeds that are even slower, barely 300kBytes/sec
How many file creates per second?
--b.
Quoted 27 lines
b) I took an external USB HDD,
- created two 20GB partitions sdc1 and sdc2
- formatted sdc1 as btrfs
- formatted sdc2 as ext4
- mounted sdc2 the same as sdb1 in test (a),
ext4, barrier=1,data=ordered
- exported both volumes over NFSunpacking a large source tarball on these two volumes, iostat reports write speeds that are around 5MB/sec - much faster than the original problem I was having
Bottom line, this leaves me with the impression that either
- the server's SATA controller or disks need a firmware upgrade,
- or there is some issue with the kernel barriers and/or cache flushing on this specific SATA hardware.I think it is fair to say that the NFS client is not at fault, however, I can imagine many people would be tempted to just use `async' when faced with a problem like this, given that async makes everything just run fast.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 45 lines · J. Bruce Fields
Quoted 44 lines · Daniel Pocock
Quoted 22 lines · Daniel Pocock
Quoted 22 lines
Quoted 9 lines
Ok, so the combination of:
- enable writeback with hdparm
- use ext4 (and not ext3)
- barrier=1 and data=writeback? or data=?- is there a particular kernel version (on either client or server side) that will offer more stability using this combination of features?
Not that I'm aware of. As long as you have a kernel > 2.6.29, then LVM should work correctly. The main problem is that some SATA hardware tends to be buggy, defeating the methods used by the barrier code to ensure data is truly on disk. I believe that XFS will therefore actually test the hardware when you mount with write caching and barriers, and should report if the test fails in the syslogs.
See http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.Quoted 3 lines
I think there are some other variations of my workflow that I can attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk because I don't need to keep the hundreds of object files.
Quoted 8 lines
Quoted 3 lines
You might also consider using something like ccache and set the
CCACHE_DIRto a local disk if you have one.Thanks for the feedback about these options, I am going to look at these strategies more closely
I decided to try and take md and LVM out of the picture, I tried two variations:
a) the boot partitions are not mirrored, so I reformatted one of them as ext4,
- enabled write-cache for the whole of sdb,
- mounted ext4, barrier=1,data=ordered
- and exported this volume over NFS
Quoted 5 lines
Quoted 2 lines
unpacking a large source tarball on this volume, iostat reports write speeds that are even slower, barely 300kBytes/sec
How many file creates per second?
I ran:
nfsstat -s -o all -l -Z5
and during the test (unpacking the tarball), I see numbers like these every 5 seconds for about 2 minutes:
nfs v3 server total: 319
------------- ------------- --------
nfs v3 server getattr: 1 nfs v3 server setattr: 126 nfs v3 server access: 6 nfs v3 server write: 61 nfs v3 server create: 61 nfs v3 server mkdir: 3 nfs v3 server commit: 61
I decided to expand the scope of my testing too, I want to rule out the possibility that my HP Microserver with onboard SATA is the culprit. I set up two other NFS servers (all Debian 6, kernel 2.6.38):
HP Z800 Xeon workstation
Intel Corporation 82801 SATA RAID Controller (operating as AHCI) VB0250EAVER (250GB 7200rpm)
Lenovo Thinkpad X220
Intel Corporation Cougar Point 6 port SATA AHCI Controller (rev 04) SSDSA2BW160G3L (160GB SSD)
Both the Z800 and X220 run as NFSv3 servers Each one has a fresh 10GB logical volume formatted ext4, mount options: barrier=1,data=ordered
write cache (hdparm -W 1): enabled
Results:
NFS client: X220
NFS server: Z800 (regular disk)
iostat reports about 1,000kbytes/sec when unpacking the tarball This is just as slow as the original NFS server
NFS client: Z800
NFS server: X220 (SSD disk)
iostat reports about 22,000kbytes/sec when unpacking the tarball
It seems that buying a pair of SSDs for my HP MicroServer will let me use NFS `sync' and enjoy healthy performance - 20x faster.
However, is there really no other way to get more speed out of NFS when using the `sync' option?
Quoted 13 lines · Daniel Pocock
Quoted 2 lines · J. Bruce Fields
How many file creates per second?
I ran:
nfsstat -s -o all -l -Z5
and during the test (unpacking the tarball), I see numbers like these every 5 seconds for about 2 minutes:nfs v3 server total: 319
------------- ------------- --------
Quoted 7 lines
nfs v3 server getattr: 1 nfs v3 server setattr: 126 nfs v3 server access: 6 nfs v3 server write: 61 nfs v3 server create: 61 nfs v3 server mkdir: 3 nfs v3 server commit: 61
OK, so it's probably creating about 60 new files, each requiring a create, write, commit, and two setattrs?
Each of those operations is synchronous, so probably has to wait for at least one disk seek. About 300 such operations every 5 seconds is about 60 per second, or about 16ms each. That doesn't sound so far off.
(I wonder why it needs two setattrs?)
Quoted 17 lines
I decided to expand the scope of my testing too, I want to rule out the possibility that my HP Microserver with onboard SATA is the culprit. I set up two other NFS servers (all Debian 6, kernel 2.6.38):
HP Z800 Xeon workstation
Intel Corporation 82801 SATA RAID Controller (operating as AHCI) VB0250EAVER (250GB 7200rpm)Lenovo Thinkpad X220
Intel Corporation Cougar Point 6 port SATA AHCI Controller (rev 04) SSDSA2BW160G3L (160GB SSD)Both the Z800 and X220 run as NFSv3 servers Each one has a fresh 10GB logical volume formatted ext4, mount options: barrier=1,data=ordered
write cache (hdparm -W 1): enabled
Quoted 5 lines
Results:
NFS client: X220
NFS server: Z800 (regular disk)
iostat reports about 1,000kbytes/sec when unpacking the tarball This is just as slow as the original NFS server
Again, reporting kbytes/second alone isn't useful--data throughput isn't interesting for a workload like unpacking a tarball with a lot of small files. The limiting factor is the synchronous operations.
Quoted 6 lines
NFS client: Z800
NFS server: X220 (SSD disk)
iostat reports about 22,000kbytes/sec when unpacking the tarballIt seems that buying a pair of SSDs for my HP MicroServer will let me use NFS `sync' and enjoy healthy performance - 20x faster.
And an SSD has much lower write latency, so this isn't surprising.
Quoted 2 lines
However, is there really no other way to get more speed out of NFS when using the `sync' option?
I've heard reports of people being able to get better performance on this kind of workload by using an external journal on an SSD.
(Last I tried this--with a machine at home, using (if I remember correctly) ext4 on software raid with the journal on an intel x25-m, I wasn't able to get any improvement. I didn't try to figure out why.)
--b.
T24gVHVlLCAyMDEyLTA1LTA4IGF0IDA4OjQ1IC0wNDAwLCBKLiBCcnVjZSBGaWVsZHMgd3JvdGU6 DQo+IE9uIFR1ZSwgTWF5IDA4LCAyMDEyIGF0IDEyOjA2OjU5UE0gKzAwMDAsIERhbmllbCBQb2Nv Y2sgd3JvdGU6DQo+ID4gDQo+ID4gDQo+ID4gT24gMDcvMDUvMTIgMTc6MTgsIEouIEJydWNlIEZp ZWxkcyB3cm90ZToNCj4gPiA+IEhvdyBtYW55IGZpbGUgY3JlYXRlcyBwZXIgc2Vjb25kPw0KPiA+ ID4gDQo+ID4gDQo+ID4gSSByYW46DQo+ID4gbmZzc3RhdCAtcyAtbyBhbGwgLWwgLVo1DQo+ID4g YW5kIGR1cmluZyB0aGUgdGVzdCAodW5wYWNraW5nIHRoZSB0YXJiYWxsKSwgSSBzZWUgbnVtYmVy cyBsaWtlIHRoZXNlDQo+ID4gZXZlcnkgNSBzZWNvbmRzIGZvciBhYm91dCAyIG1pbnV0ZXM6DQo+ ID4gDQo+ID4gbmZzIHYzIHNlcnZlciAgICAgICAgdG90YWw6ICAgICAgMzE5DQo+ID4gLS0tLS0t LS0tLS0tLSAtLS0tLS0tLS0tLS0tIC0tLS0tLS0tDQo+ID4gbmZzIHYzIHNlcnZlciAgICAgIGdl dGF0dHI6ICAgICAgICAxDQo+ID4gbmZzIHYzIHNlcnZlciAgICAgIHNldGF0dHI6ICAgICAgMTI2 DQo+ID4gbmZzIHYzIHNlcnZlciAgICAgICBhY2Nlc3M6ICAgICAgICA2DQo+ID4gbmZzIHYzIHNl cnZlciAgICAgICAgd3JpdGU6ICAgICAgIDYxDQo+ID4gbmZzIHYzIHNlcnZlciAgICAgICBjcmVh dGU6ICAgICAgIDYxDQo+ID4gbmZzIHYzIHNlcnZlciAgICAgICAgbWtkaXI6ICAgICAgICAzDQo+ ID4gbmZzIHYzIHNlcnZlciAgICAgICBjb21taXQ6ICAgICAgIDYxDQo+IA0KPiBPSywgc28gaXQn cyBwcm9iYWJseSBjcmVhdGluZyBhYm91dCA2MCBuZXcgZmlsZXMsIGVhY2ggcmVxdWlyaW5nIGEN Cj4gY3JlYXRlLCB3cml0ZSwgY29tbWl0LCBhbmQgdHdvIHNldGF0dHJzPw0KPiANCj4gRWFjaCBv ZiB0aG9zZSBvcGVyYXRpb25zIGlzIHN5bmNocm9ub3VzLCBzbyBwcm9iYWJseSBoYXMgdG8gd2Fp dCBmb3IgYXQNCj4gbGVhc3Qgb25lIGRpc2sgc2Vlay4gIEFib3V0IDMwMCBzdWNoIG9wZXJhdGlv bnMgZXZlcnkgNSBzZWNvbmRzIGlzIGFib3V0DQo+IDYwIHBlciBzZWNvbmQsIG9yIGFib3V0IDE2 bXMgZWFjaC4gIFRoYXQgZG9lc24ndCBzb3VuZCBzbyBmYXIgb2ZmLg0KPiANCj4gKEkgd29uZGVy IHdoeSBpdCBuZWVkcyB0d28gc2V0YXR0cnM/KQ0KDQpJdCBpcyBhbiB1bnRhciB3b3JrbG9hZCwg c28gaXQgbmVlZHMgdG8gcmVzZXQgdGhlIGF0aW1lL210aW1lIGFmdGVyDQp3cml0aW5nIHRoZSBm aWxlLg0KDQpOb3RlIHRoYXQgdGhlIGFib3ZlIHdvcmtsb2FkIGlzIGV4YWN0bHkgdGhlIG9uZSBJ IHdhcyB0YXJnZXRpbmcgd2l0aCB0aGUNCnVuc3RhYmxlIGZpbGUgY3JlYXRpb24gZHJhZnQ6DQoN Cmh0dHA6Ly90b29scy5pZXRmLm9yZy9odG1sL2RyYWZ0LW15a2xlYnVzdC1uZnN2NC11bnN0YWJs ZS1maWxlLWNyZWF0aW9uLTAxDQoNCkknbSBnb2luZyB0byB0cnkgcHVzaGluZyB0aGF0IGFnYWlu IGZvciBORlN2NC4zLi4uDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50 IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5l dGFwcC5jb20NCg0K
Quoted 22 lines · J. Bruce Fields
Quoted 13 lines · Daniel Pocock
Quoted 2 lines · J. Bruce Fields
How many file creates per second?
I ran:
nfsstat -s -o all -l -Z5
and during the test (unpacking the tarball), I see numbers like these every 5 seconds for about 2 minutes:nfs v3 server total: 319
------------- ------------- --------Quoted 7 lines
nfs v3 server getattr: 1 nfs v3 server setattr: 126 nfs v3 server access: 6 nfs v3 server write: 61 nfs v3 server create: 61 nfs v3 server mkdir: 3 nfs v3 server commit: 61
Quoted 8 lines
OK, so it's probably creating about 60 new files, each requiring a create, write, commit, and two setattrs?
Each of those operations is synchronous, so probably has to wait for at least one disk seek. About 300 such operations every 5 seconds is about 60 per second, or about 16ms each. That doesn't sound so far off.
(I wonder why it needs two setattrs?)
I checked that with wireshark:
- first SETATTR sets mode=0644 and atime=mtime=`set to server time'
- second SETATTR sets atime and mtime using client values (atime=now, mtime= some time in the past)
Is this likely to be an application issue (e.g. in tar), or should the NFS client be able to merge those two requests somehow?
If I add `m' to my tar command (tar xzmf) the nfsstat results change:
nfs v3 server total: 300
------------- ------------- --------
nfs v3 server setattr: 82 nfs v3 server lookup: 17 nfs v3 server access: 5 nfs v3 server write: 90 nfs v3 server create: 83 nfs v3 server mkdir: 3 nfs v3 server remove: 15 nfs v3 server commit: 5
and iostat reports that w/s is about the same (290/sec) but throughput is now in the region 1.1 - 1.5MBytes/sec
Without using `m', here are the full stats from the Z800 acting as NFSv3 server:
nfs v3 server total: 299
------------- ------------- --------
nfs v3 server setattr: 132 nfs v3 server access: 6 nfs v3 server write: 88 nfs v3 server create: 62 nfs v3 server mkdir: 6 nfs v3 server commit: 5
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
dm-10 0.00 0.00 0.00 294.00 0.00 1.15
8.00 1.04 3.55 3.26 95.92
The other thing that stands out is this:
avgrq-sz = 8 (units = 512 byte sectors)
If the server filesystem is btrfs, the wMB/s figure is the same, but I notice avgrq-sz = 16, so it seems to be combining more requests into bigger writes
Here is the entry from /proc/mounts:/dev/mapper/vg01-test1 /mnt/test1 ext4
rw,relatime,barrier=1,data=ordered 0 0