Skip to contentSkip to conversation

Export over NFS sets rsize to 1MB?

Lustre-devel James Vanns 3 participants 5 messages May 13, 2013 – May 15, 2013 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hello dev list. Apologies for a post to perhaps the wrong group but I'm having a bit of difficulty locating any document or wiki describing how and/or where the preferred read and write block size for NFS exports of a Lustre filesystem are set to 1MB?

Basically we have two Lustre filesystems exported over NFSv3. Our lustre block size is 4k and the max r/w size is 1MB. Without any special rsize/wsize options set for the export the default one suggested to clients (MOUNT->FSINFO RPC) as the preferred size is set to 1MB. How does Lustre figure this out? Other non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes.

Any hints would be appreciated. Documentation or code paths welcome as are annotated /proc locations.

Thanks,

Jim

PS. This is Lustre 1.8.8 on Linux 2.6.32

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#2 lore raw
Quoted 7 lines · James Vanns

Hello dev list. Apologies for a post to perhaps the wrong group but I'm having a
bit of difficulty locating any document or wiki describing how and/or where the
preferred read and write block size for NFS exports of a Lustre filesystem are
set to 1MB?

1MB is the RPC size and "optimal IO size" for Lustre. This would normally be exported to applications via the stat(2) "st_blksize" field, though it is typically 2MB (2x the RPC size in order to allow some pipelining). I suspect this is where NFS is getting the value, since it is not passed up via the statfs(2) call.

Quoted 8 lines

Basically we have two Lustre filesystems exported over NFSv3. Our lustre block size
is 4k and the max r/w size is 1MB. Without any special rsize/wsize options set for
the export the default one suggested to clients (MOUNT->FSINFO RPC) as the preferred
size is set to 1MB. How does Lustre figure this out? Other non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes.

Taking a quick look at the code, it looks like NFS TCP connections all have a maximum max_payload of 1MB, but this is limited in a number of places in the code by the actual read size, and other maxima (for which I can't easily find the source value).

Quoted 2 lines

Any hints would be appreciated. Documentation or code paths welcome as are annotated /proc locations.

To clarify from your question - is this large blocksize causing a performance problem? I recall some applications having problems with stdio "fread()" and friends reading too much data into their buffers if they are doing random IO. Ideally stdio shouldn't be reading more than it needs when doing random IO.

At one time in the past, we derived the st_blksize from the file stripe_size, but this caused problems with the NFS "Connectathon" or similar. It is currently limited by LL_MAX_BLKSIZE_BITS for all files, but I wouldn't recommend reducing this directly, since it would also affect "cp" and others that also depend on st_blksize for the "optimal IO size". It would be possible to reintroduce the per-file tunable in ll_update_inode() I think.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#3 lore raw

Thanks for replying Andreas...

Quoted 12 lines
Quoted 9 lines · James Vanns

Hello dev list. Apologies for a post to perhaps the wrong group but I'm
having a
bit of difficulty locating any document or wiki describing how and/or
where the
preferred read and write block size for NFS exports of a Lustre filesystem are
set to 1MB?

Quoted 9 lines

1MB is the RPC size and "optimal IO size" for Lustre. This would normally
be exported to applications via the stat(2) "st_blksize" field, though it
is typically 2MB (2x the RPC size in order to allow some pipelining).
I
suspect this is where NFS is getting the value, since it is not passed up
via the statfs(2) call.

Hmm. OK. I've confirmed it isn't from any struct stat{} attribute (st_blksize is still just 4k) but yes, our RPC size is 1MB. It isn't coming from statfs() or statvfs() either.

Quoted 12 lines
Quoted 11 lines

Basically we have two Lustre filesystems exported over NFSv3. Our lustre
block size
is 4k and the max r/w size is 1MB. Without any special rsize/wsize options set for
the export the default one suggested to clients (MOUNT->FSINFO RPC) as
the preferred
size is set to 1MB. How does Lustre figure this out? Other non-Lustre
exports are generally much less; 4, 8, 16 or 32 kilobytes.

Quoted 6 lines

Taking a quick look at the code, it looks like NFS TCP connections all
have a maximum max_payload of 1MB, but this is limited in a number of places in the code by the actual read size, and other maxima (for which I
can't easily find the source value).

Yes it seems that 1MB is the maximum but also the optimal or preferred.

Quoted 11 lines
Quoted 3 lines

Any hints would be appreciated. Documentation or code paths welcome as
are annotated /proc locations.

To clarify from your question - is this large blocksize causing a performance problem? I recall some applications having problems with stdio "fread()" and friends reading too much data into their buffers if
they are doing random IO. Ideally stdio shouldn't be reading more than it
needs when doing random IO.

We're experiencing what appears to be (as of yet I have no hard evidence) contention due to connection 'hogging' for these large reads. We have a set of 4 NFS servers in a DNS round-robin all configured to serve up our Lustre filesystem across 64 knfsds (per host). It's possible that we simply don't have enough hosts (or knfsds) for the #clients because many of the clients will be reading large amounts of data (1MB at a time) and therefore preventing other queued clients from getting a look-in. Of course this appears to the user as just a very slow experience.

At the moment, I'm just trying to understand where this 1MB is coming from! The RPC transport size (I forgot to confirm - yes, we're serving NFS over TCP) is 1MB for all other 'regular' NFS servers yet their r/wsize are quite different.

Thanks for the feedback and sorry I can't be more accurate at the moment :\

Jim

Quoted 19 lines

At one time in the past, we derived the st_blksize from the file stripe_size, but this caused problems with the NFS "Connectathon" or similar. It is currently limited by LL_MAX_BLKSIZE_BITS for all files,
but I wouldn't recommend reducing this directly, since it would also affect "cp" and others that also depend on st_blksize for the "optimal IO
size". It would be possible to reintroduce the per-file tunable in ll_update_inode() I think.

Cheers, Andreas
--
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#4 lore raw
Quoted 13 lines · James Vanns
Quoted 7 lines
Quoted 4 lines · James Vanns

Hello dev list. Apologies for a post to perhaps the wrong group but I'm having a bit of difficulty locating any document or wiki describing how and/or where the preferred read and write block size for NFS exports of a Lustre filesystem are set to 1MB?

Quoted 5 lines

1MB is the RPC size and "optimal IO size" for Lustre. This would normally be exported to applications via the stat(2) "st_blksize" field, though it is typically 2MB (2x the RPC size in order to allow some pipelining). I suspect this is where NFS is getting the value, since it is not passed up via the statfs(2) call.

Quoted 5 lines

Hmm. OK. I've confirmed it isn't from any struct stat{} attribute (st_blksize
is still just 4k) but yes, our RPC size is 1MB. It isn't coming from statfs()
or statvfs() either.

I've CC'd the Linux NFS mailing list, since I don't know enough about the NFS client/server code to decide where this is coming from either.

James, what kernel version do you have on the Lustre clients (NFS servers) and on the NFS clients?

Quoted 36 lines
Quoted 11 lines
Quoted 6 lines

Basically we have two Lustre filesystems exported over NFSv3. Our lustre block size is 4k and the max r/w size is 1MB. Without any special rsize/wsize options set for the export the default one suggested to clients (MOUNT->FSINFO RPC) as the preferred size is set to 1MB. How does Lustre figure this out? Other non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes.

Taking a quick look at the code, it looks like NFS TCP connections all have a maximum max_payload of 1MB, but this is limited in a number of places in the code by the actual read size, and other maxima (for which I can't easily find the source value).

Yes it seems that 1MB is the maximum but also the optimal or preferred.

Quoted 8 lines
Quoted 2 lines

Any hints would be appreciated. Documentation or code paths welcome as are annotated /proc locations.

To clarify from your question - is this large blocksize causing a performance problem? I recall some applications having problems with stdio "fread()" and friends reading too much data into their buffers if they are doing random IO. Ideally stdio shouldn't be reading more than it needs when doing random IO.

We're experiencing what appears to be (as of yet I have no hard evidence) contention due to connection 'hogging' for these large reads. We have a set
of 4 NFS servers in a DNS round-robin all configured to serve up our Lustre
filesystem across 64 knfsds (per host). It's possible that we simply don't have enough hosts (or knfsds) for the #clients because many of the clients will be reading large amounts of data (1MB at a time) and therefore preventing
other queued clients from getting a look-in. Of course this appears to the user
as just a very slow experience.

Quoted 8 lines

At the moment, I'm just trying to understand where this 1MB is coming from!
The RPC transport size (I forgot to confirm - yes, we're serving NFS over TCP) is 1MB for all other 'regular' NFS servers yet their r/wsize are quite different.

Thanks for the feedback and sorry I can't be more accurate at the moment :\

It should also be possible to explicitly mount the clients with rsize=65536 and wsize=65536, but it would be better to understand the cause of this.

Quoted 8 lines
Quoted 8 lines

At one time in the past, we derived the st_blksize from the file stripe_size, but this caused problems with the NFS "Connectathon" or similar because the block size would change from when the file was first opened. It is currently limited by LL_MAX_BLKSIZE_BITS for all files, but I wouldn't recommend reducing this directly, since it would also affect "cp" and others that also depend on st_blksize for the "optimal IO size". It would be possible to reintroduce the per-file tunable in ll_update_inode() I think.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#5 lore raw

Thanks for the help Andreas - I too asked the NFS developers (on the list) and they came back with /proc/fs/nfsd/max_block_size and nfsd_get_default_max_blksize() in fs/nfsd/nfssvc.c.

For what it's worth, this is kernel 2.6.32 and Lustre 1.8.8.

Reckon I've got what I asked for now ;)

Jim

----- Original Message -----
From: "Andreas Dilger" <[email protected]>
To: "james vanns" <[email protected]>
Cc: lustre-devel at lists.lustre.org, linux-nfs at vger.kernel.org
Sent: Tuesday, 14 May, 2013 11:06:08 PM
Subject: Re: [Lustre-devel] Export over NFS sets rsize to 1MB?

Quoted 13 lines · James Vanns
Quoted 7 lines
Quoted 4 lines · James Vanns

Hello dev list. Apologies for a post to perhaps the wrong group but I'm having a bit of difficulty locating any document or wiki describing how and/or where the preferred read and write block size for NFS exports of a Lustre filesystem are set to 1MB?

Quoted 5 lines

1MB is the RPC size and "optimal IO size" for Lustre. This would normally be exported to applications via the stat(2) "st_blksize" field, though it is typically 2MB (2x the RPC size in order to allow some pipelining). I suspect this is where NFS is getting the value, since it is not passed up via the statfs(2) call.

Quoted 5 lines

Hmm. OK. I've confirmed it isn't from any struct stat{} attribute (st_blksize
is still just 4k) but yes, our RPC size is 1MB. It isn't coming from statfs()
or statvfs() either.

I've CC'd the Linux NFS mailing list, since I don't know enough about the NFS client/server code to decide where this is coming from either.

James, what kernel version do you have on the Lustre clients (NFS servers) and on the NFS clients?

Quoted 36 lines
Quoted 11 lines
Quoted 6 lines

Basically we have two Lustre filesystems exported over NFSv3. Our lustre block size is 4k and the max r/w size is 1MB. Without any special rsize/wsize options set for the export the default one suggested to clients (MOUNT->FSINFO RPC) as the preferred size is set to 1MB. How does Lustre figure this out? Other non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes.

Taking a quick look at the code, it looks like NFS TCP connections all have a maximum max_payload of 1MB, but this is limited in a number of places in the code by the actual read size, and other maxima (for which I can't easily find the source value).

Yes it seems that 1MB is the maximum but also the optimal or preferred.

Quoted 8 lines
Quoted 2 lines

Any hints would be appreciated. Documentation or code paths welcome as are annotated /proc locations.

To clarify from your question - is this large blocksize causing a performance problem? I recall some applications having problems with stdio "fread()" and friends reading too much data into their buffers if they are doing random IO. Ideally stdio shouldn't be reading more than it needs when doing random IO.

We're experiencing what appears to be (as of yet I have no hard evidence) contention due to connection 'hogging' for these large reads. We have a set
of 4 NFS servers in a DNS round-robin all configured to serve up our Lustre
filesystem across 64 knfsds (per host). It's possible that we simply don't have enough hosts (or knfsds) for the #clients because many of the clients will be reading large amounts of data (1MB at a time) and therefore preventing
other queued clients from getting a look-in. Of course this appears to the user
as just a very slow experience.

Quoted 8 lines

At the moment, I'm just trying to understand where this 1MB is coming from!
The RPC transport size (I forgot to confirm - yes, we're serving NFS over TCP) is 1MB for all other 'regular' NFS servers yet their r/wsize are quite different.

Thanks for the feedback and sorry I can't be more accurate at the moment :\

It should also be possible to explicitly mount the clients with rsize=65536 and wsize=65536, but it would be better to understand the cause of this.

Quoted 8 lines
Quoted 8 lines

At one time in the past, we derived the st_blksize from the file stripe_size, but this caused problems with the NFS "Connectathon" or similar because the block size would change from when the file was first opened. It is currently limited by LL_MAX_BLKSIZE_BITS for all files, but I wouldn't recommend reducing this directly, since it would also affect "cp" and others that also depend on st_blksize for the "optimal IO size". It would be possible to reintroduce the per-file tunable in ll_update_inode() I think.

Cheers, Andreas

Signature · 15 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division


--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Jim Vanns
Senior Software Developer
Framestore