pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout
Subsystems (2)
In the recent pnfs kernel versions, WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout.
In nfs_write_rpcsetup function, the value of the variable 'stable' will always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT flag will not be set.
The value of the variable ncommit' will be ++ in the later process (nfs_writeback_release_common function).
Thanks!
Quoted 2 lines · chenditang
In the recent pnfs kernel versions, WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout.
I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS and that the DS is returning how=FILE_SYNC and that the client is implementing the behavior described in Errata ID: 2751
http://www.rfc-editor.org/errata_search.php?rfc=5661
Do you have a tcp dump of client, MDS, and DS traffic?
Quoted 4 lines
In
nfs_write_rpcsetupfunction, the value of the variable 'stable' will always beNFS_FILE_SYNC(ncommit is 0). soNFS_INO_LAYOUTCOMMITflag will not be set.
What's the value of the "how" arg to nfs_write_rpcsetup that you see? At any rate, if you look at filelayout_set_layoutcommit, it will not call pnfs_set_layoutcommit in the following cases:
if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
wdata->res.verf->committed == NFS_FILE_SYNC)
return;Benny
Quoted 7 lines
The value of the variable ncommit' will be ++ in the later process (
nfs_writeback_release_commonfunction).Thanks!
Quoted 9 lines · Benny Halevy
Quoted 2 lines · chenditang
In the recent pnfs kernel versions, WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout.
I presume that the file server sets
NFL4_UFLG_COMMIT_THRU_MDSand that the DS is returning how=FILE_SYNC and that the client is implementing the behavior described in Errata ID: 2751
http://www.rfc-editor.org/errata_search.php?rfc=5661Do you have a tcp dump of client, MDS, and DS traffic?
NFL4_UFLG_COMMIT_THRU_MDS is not set, "nfl_util" is 8192 and "commit_through_mds" is 0 to filelayout_decode_layout.
Quoted 10 lines
Quoted 3 lines
In
nfs_write_rpcsetupfunction, the value of the variable 'stable' will always beNFS_FILE_SYNC(ncommit is 0). soNFS_INO_LAYOUTCOMMITflag will not be set.What's the value of the "how" arg to
nfs_write_rpcsetupthat you see? At any rate, if you look atfilelayout_set_layoutcommit, it will not callpnfs_set_layoutcommitin the following cases:if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds || wdata->res.verf->committed == NFS_FILE_SYNC) return;
"how" is 32, and "cinfo->mds->ncommit" is 0. pnfs_set_layoutcommit has not beeb called. I think "cinfo->mds->ncommit" can not be used here.
I dump the stack, and the initial value of "ncommit" is 0, then it to be used in nfs_write_rpcsetup function, but "ncommit++" in nfs_writeback_release_common function(if NFS_FILE_SYNC is not set).
static const struct rpc_call_ops nfs_write_common_ops = { .rpc_call_prepare = nfs_write_prepare, .rpc_call_done = nfs_writeback_done_common, .rpc_release = nfs_writeback_release_common,
};
The code processes:
step1: init_once --> nfsi->commit_info.ncommit = 0;
setp2: nfs_flush_one --> nfs_write_rpcsetup
setp3: nfs_writeback_done --> nfs_writeback_done --> filelayout_write_done_cb --> filelayout_set_layoutcommit
setp4: nfs_writeback_release_common--> ...-->nfs_write_completion --> .... --> cinfo->mds->ncommit++;
Quoted 10 lines
Benny
Quoted 6 lines
The value of the variable ncommit' will be ++ in the later process (
nfs_writeback_release_commonfunction).Thanks!
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3duZXJAdmdl ci5rZXJuZWwub3JnIFttYWlsdG86bGludXgtbmZzLQ0KPiBvd25lckB2Z2VyLmtlcm5lbC5vcmdd IE9uIEJlaGFsZiBPZiBjaGVuZGl0YW5nDQo+IFNlbnQ6IE1vbmRheSwgTWFyY2ggMTgsIDIwMTMg MzoxMiBBTQ0KPiBUbzogYmhhbGV2eUB0b25pYW4uY29tDQo+IENjOiBsaW51eC1uZnNAdmdlci5r ZXJuZWwub3JnOyBMSU5FDQo+IFN1YmplY3Q6IHBuZnM6IFdSSVRFIG9wZXJhdGlvbiB3aWxsIG5v dCBzZW5kIExBWU9VVENPTU1JIE9QIHRvIE1EUyBpZg0KPiB1c2UgZmlsZWxheW91dA0KPiANCj4g SW4gdGhlIHJlY2VudCBwbmZzIGtlcm5lbCB2ZXJzaW9ucywgV1JJVEUgb3BlcmF0aW9uIHdpbGwg bm90IHNlbmQNCj4gTEFZT1VUQ09NTUkgT1AgdG8gTURTIGlmIHVzZSBmaWxlbGF5b3V0Lg0KPiAN Cj4gSW4gbmZzX3dyaXRlX3JwY3NldHVwIGZ1bmN0aW9uLCB0aGUgdmFsdWUgb2YgdGhlIHZhcmlh YmxlICdzdGFibGUnIHdpbGwgYWx3YXlzDQo+IGJlIE5GU19GSUxFX1NZTkMobmNvbW1pdCBpcyAw KS4gc28gTkZTX0lOT19MQVlPVVRDT01NSVQgZmxhZyB3aWxsIG5vdA0KPiBiZSBzZXQuDQoNCklm IHRoZXNlIGFyZSBzbWFsbCBmaWxlcywgdGhlbiB0aGF0IGlzIGV4cGVjdGVkIGJlaGF2aW91ci4g VGhlIHBORlMgc3BlYyBkb2VzIG5vdCBleGVtcHQgdGhlIERTIGZyb20gaGF2aW5nIHRvIGhvbm91 ciB0aGUgJ3N0YWJsZScgZmxhZy4NCg0KQ2hlZXJzDQogIFRyb25kDQo=