Skip to contentSkip to conversation

pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

DISCUSSION chenditang 3 participants 4 messages Mar 18, 2013

Subsystems (2)
lore
Share & tools
lore

chenditang

author
#0 lore raw

In the recent pnfs kernel versions, WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout.

In nfs_write_rpcsetup function, the value of the variable 'stable' will always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT flag will not be set.

The value of the variable ncommit' will be ++ in the later process (nfs_writeback_release_common function).

Thanks!

#1 lore raw
Quoted 2 lines · chenditang

In the recent pnfs kernel versions, WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout.

I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS and that the DS is returning how=FILE_SYNC and that the client is implementing the behavior described in Errata ID: 2751
http://www.rfc-editor.org/errata_search.php?rfc=5661

Do you have a tcp dump of client, MDS, and DS traffic?

Quoted 4 lines

In nfs_write_rpcsetup function, the value of the variable 'stable' will always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT flag will not be set.

What's the value of the "how" arg to nfs_write_rpcsetup that you see? At any rate, if you look at filelayout_set_layoutcommit, it will not call pnfs_set_layoutcommit in the following cases:

        if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
            wdata->res.verf->committed == NFS_FILE_SYNC)
                return;

Benny

Quoted 7 lines

The value of the variable ncommit' will be ++ in the later process (nfs_writeback_release_common function).

Thanks!

author
#2 lore raw
Quoted 9 lines · Benny Halevy
Quoted 2 lines · chenditang

In the recent pnfs kernel versions, WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout.

I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS and that the DS is returning how=FILE_SYNC and that the client is implementing the behavior described in Errata ID: 2751
http://www.rfc-editor.org/errata_search.php?rfc=5661

Do you have a tcp dump of client, MDS, and DS traffic?

NFL4_UFLG_COMMIT_THRU_MDS is not set, "nfl_util" is 8192 and "commit_through_mds" is 0 to filelayout_decode_layout.

Quoted 10 lines
Quoted 3 lines

In nfs_write_rpcsetup function, the value of the variable 'stable' will always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT flag will not be set.

What's the value of the "how" arg to nfs_write_rpcsetup that you see? At any rate, if you look at filelayout_set_layoutcommit, it will not call pnfs_set_layoutcommit in the following cases:

         if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
             wdata->res.verf->committed == NFS_FILE_SYNC)
                 return;

"how" is 32, and "cinfo->mds->ncommit" is 0. pnfs_set_layoutcommit has not beeb called. I think "cinfo->mds->ncommit" can not be used here.

I dump the stack, and the initial value of "ncommit" is 0, then it to be used in nfs_write_rpcsetup function, but "ncommit++" in nfs_writeback_release_common function(if NFS_FILE_SYNC is not set).

static const struct rpc_call_ops nfs_write_common_ops = { .rpc_call_prepare = nfs_write_prepare, .rpc_call_done = nfs_writeback_done_common, .rpc_release = nfs_writeback_release_common,
};

The code processes:
step1: init_once --> nfsi->commit_info.ncommit = 0;
setp2: nfs_flush_one --> nfs_write_rpcsetup
setp3: nfs_writeback_done --> nfs_writeback_done --> filelayout_write_done_cb --> filelayout_set_layoutcommit
setp4: nfs_writeback_release_common--> ...-->nfs_write_completion --> .... --> cinfo->mds->ncommit++;

Quoted 10 lines

Benny

Quoted 6 lines

The value of the variable ncommit' will be ++ in the later process (nfs_writeback_release_common function).

Thanks!

#3 lore raw
In reply to chenditang

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3duZXJAdmdl ci5rZXJuZWwub3JnIFttYWlsdG86bGludXgtbmZzLQ0KPiBvd25lckB2Z2VyLmtlcm5lbC5vcmdd IE9uIEJlaGFsZiBPZiBjaGVuZGl0YW5nDQo+IFNlbnQ6IE1vbmRheSwgTWFyY2ggMTgsIDIwMTMg MzoxMiBBTQ0KPiBUbzogYmhhbGV2eUB0b25pYW4uY29tDQo+IENjOiBsaW51eC1uZnNAdmdlci5r ZXJuZWwub3JnOyBMSU5FDQo+IFN1YmplY3Q6IHBuZnM6IFdSSVRFIG9wZXJhdGlvbiB3aWxsIG5v dCBzZW5kIExBWU9VVENPTU1JIE9QIHRvIE1EUyBpZg0KPiB1c2UgZmlsZWxheW91dA0KPiANCj4g SW4gdGhlIHJlY2VudCBwbmZzIGtlcm5lbCB2ZXJzaW9ucywgV1JJVEUgb3BlcmF0aW9uIHdpbGwg bm90IHNlbmQNCj4gTEFZT1VUQ09NTUkgT1AgdG8gTURTIGlmIHVzZSBmaWxlbGF5b3V0Lg0KPiAN Cj4gSW4gbmZzX3dyaXRlX3JwY3NldHVwIGZ1bmN0aW9uLCB0aGUgdmFsdWUgb2YgdGhlIHZhcmlh YmxlICdzdGFibGUnIHdpbGwgYWx3YXlzDQo+IGJlIE5GU19GSUxFX1NZTkMobmNvbW1pdCBpcyAw KS4gc28gTkZTX0lOT19MQVlPVVRDT01NSVQgZmxhZyB3aWxsIG5vdA0KPiBiZSBzZXQuDQoNCklm IHRoZXNlIGFyZSBzbWFsbCBmaWxlcywgdGhlbiB0aGF0IGlzIGV4cGVjdGVkIGJlaGF2aW91ci4g VGhlIHBORlMgc3BlYyBkb2VzIG5vdCBleGVtcHQgdGhlIERTIGZyb20gaGF2aW5nIHRvIGhvbm91 ciB0aGUgJ3N0YWJsZScgZmxhZy4NCg0KQ2hlZXJzDQogIFRyb25kDQo=