state manager failed on NFSv4 server

A kernel built from Benny's pnfs-all-latest can no longer mount the EMC server because nfs4_state_manager() fails. I suspect it doesn't like something about the EXCHANGE_ID reply. This worked before Benny's last update (late Dec?), and I can bisect if necessary, but wondered if someone here can take a look and see if there is something obviously wrong.

I put a dump of the call and reply here:
http://www.citi.umich.edu/projects/nfsv4/pnfs/block/emc-ng-state-mgr

I can of course provide more info if that would help.

That is correct. According to your wireshark dump, your server is returning an illegal combination of the EXCHGID4_FLAG_USE_* flags.

The combination

(EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS)

is not listed in the table in section 13.1 of RFC5661 as an acceptable return value.

Cheers
Trond

Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

Trond Myklebust wrote:

The combination

(EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS)

is not listed in the table in section 13.1 of RFC5661 as an acceptable return value.

But section 13 doesn't apply to the block layout server, does it?

If you are a block layout server, the only valid return value is

EXCHGID4_FLAG_USE_PNFS_MDS

'cos you are clearly not a non-metadata server, and you clearly can't be a data server.

Cheers
Trond

Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

If you are a block layout server, the only valid return value is

EXCHGID4_FLAG_USE_PNFS_MDS

'cos you are clearly not a non-metadata server, and you clearly can't be a data server.

Sorry to be dense, this is the first time I've thought about this.

At the time of the EXCHANGE_ID call, how is the server supposed to know what kind of layout is going to be negotiated? It doesn't yet know whether the client is even going to ask for a layout, does it?

It doesn't matter. EXCHGID4_FLAG_USE_PNFS_MDS is the server's way of advertising to the client that it supports LAYOUTGET and other pNFS related operations. The client is free to ignore that message if it so desires, and just use read/write through MDS. The point here is to tell the client whether or not it should try pNFS if it can.

Cheers
Trond

Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

It doesn't matter. EXCHGID4_FLAG_USE_PNFS_MDS is the server's way of advertising to the client that it supports LAYOUTGET and other pNFS related operations. The client is free to ignore that message if it so desires, and just use read/write through MDS. The point here is to tell the client whether or not it should try pNFS if it can.

I guess that makes sense. But I think section 13 is a bit misleading since it contains requirements for the block layout too.

13. NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type

This section describes the semantics and format of NFSv4.1 file-based layouts for pNFS. NFSv4.1 file-based layouts use the LAYOUT4_NFSV4_1_FILES layout type. The LAYOUT4_NFSV4_1_FILES type defines striping data across multiple NFSv4.1 data servers.

True, but section 18.35. Operation 42: EXCHANGE_ID - Instantiate Client ID specifically says:

The EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_MDS, and EXCHGID4_FLAG_USE_PNFS_DS bits are described in Section 13.1 and convey roles the client ID is to be used for in a pNFS environment. The server MUST set one of the acceptable combinations of these bits (roles) in eir_flags, as specified in Section 13.1.

Benny

Hi Jim

The EMC server is returning exchange_id flags = 0x00070000 which is illegal - it's claming the NON_PNFS, PNFS and DS roles.

#define EXCHGID4_FLAG_USE_NON_PNFS              0x00010000
#define EXCHGID4_FLAG_USE_PNFS_MDS              0x00020000
#define EXCHGID4_FLAG_USE_PNFS_DS               0x00040000
#define EXCHGID4_FLAG_MASK_PNFS                 0x00070000

See Section 13.1 of RFC 5661

-->Andy