Does "sync" cause the FUA bit to be set?

Hi,

we have a raid controller serving a file system that we export and mount with NFS. Depending on the performance profile we chose for the raid controller, the are huge performance differences on the NFS client side between two profiles. Those two profiles both use write cache and no journaling, but one honors the FUA bit if it is set, and the other profile ignores it (there might be more differences, but the manual doesn't state any others but the FUA bit).

With the profile ignoring the FUA bit, copying or deleting directories of e.g. 10M with a about 1000 files is factor 5 faster than with the profile honoring the FUA bit.

We export with the "sync" option. Does that option maybe set the FUA bit for all write operations on the NFS server?

cu,
Frank

With the profile ignoring the FUA bit, copying or deleting directories of e.g. 10M with a about 1000 files is factor 5 faster than with the profile honoring the FUA bit.

We export with the "sync" option. Does that option maybe set the FUA bit for all write operations on the NFS server?

Quoted 17 lines · Wendy Cheng

Quoted 5 lines · Frank Steiner

With the profile ignoring the FUA bit, copying or deleting directories of e.g. 10M with a about 1000 files is factor 5 faster than with the profile honoring the FUA bit.

FUA bit is normally combined with write-thru scsi command that bypasses storage write cache. I would imagine it needs to well synchronize various pieces before issuing this command. It could hurt the performance if not done well, particularly for meta data. So your result is not surprising.

Quoted 4 lines

We export with the "sync" option. Does that option maybe set the FUA bit for all write operations on the NFS server?

Quoted 8 lines

It depends on how the filesystem (and its associated disk subsystem) is implemented. The "sync" export option itself has a heavy performance impact, regardless how FUA bit is handled. Some vendors uses specialized HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem doesn't have this type of support, you should expect "sync" option runs much much slower than "async". It is a choice (or balance) between cost, performance, and data reliability.

Wendy, I *think* what you have in mind is the sync mount option rather than the sync export flag. The latter just tells the server not to cheat and do everything asynchronously. It should *not* have a heavy performance penalty for I/O intensive writes if the client is using async writes and commits.

Quoted 28 lines · Benny Halevy

Quoted 17 lines · Wendy Cheng

Quoted 5 lines · Frank Steiner

With the profile ignoring the FUA bit, copying or deleting directories of e.g. 10M with a about 1000 files is factor 5 faster than with the profile honoring the FUA bit.

FUA bit is normally combined with write-thru scsi command that bypasses storage write cache. I would imagine it needs to well synchronize various pieces before issuing this command. It could hurt the performance if not done well, particularly for meta data. So your result is not surprising.

Quoted 4 lines

We export with the "sync" option. Does that option maybe set the FUA bit for all write operations on the NFS server?

Quoted 8 lines

It depends on how the filesystem (and its associated disk subsystem) is implemented. The "sync" export option itself has a heavy performance impact, regardless how FUA bit is handled. Some vendors uses specialized HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem doesn't have this type of support, you should expect "sync" option runs much much slower than "async". It is a choice (or balance) between cost, performance, and data reliability.

Quoted 7 lines

Wendy, I *think* what you have in mind is the sync mount option rather than the sync export flag. The latter just tells the server not to cheat and do everything asynchronously. It should *not* have a heavy performance penalty for I/O intensive writes if the client is using async writes and commits.

No, I didn't get confused ... We can use Linux as an example :) .. check out:
http://marc.info/?l=linux-nfs&m=119618886105337&w=2

-- quote

The default export might have been "async", but unless the option "sync" in /etc/exports was being ignored I was already using "sync". Nevertheless I will try to change to async and test if it makes a difference.

(one day later: )

I have now tried it and the load on the NFS server is much lower and KDE logins seem to be reasonably fast now.

Quoted 61 lines · Wendy Cheng

Quoted 33 lines · Benny Halevy

Quoted 21 lines · Wendy Cheng

Quoted 6 lines · Frank Steiner

With the profile ignoring the FUA bit, copying or deleting directories of e.g. 10M with a about 1000 files is factor 5 faster than with the profile honoring the FUA bit.

FUA bit is normally combined with write-thru scsi command that bypasses storage write cache. I would imagine it needs to well synchronize various pieces before issuing this command. It could hurt the performance if not done well, particularly for meta data. So your result is not surprising.

Quoted 5 lines

We export with the "sync" option. Does that option maybe set the FUA bit for all write operations on the NFS server?

Quoted 9 lines

It depends on how the filesystem (and its associated disk subsystem) is implemented. The "sync" export option itself has a heavy performance impact, regardless how FUA bit is handled. Some vendors uses specialized HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem doesn't have this type of support, you should expect "sync" option runs much much slower than "async". It is a choice (or balance) between cost, performance, and data reliability.

Quoted 8 lines

Wendy, I *think* what you have in mind is the sync mount option rather than the sync export flag. The latter just tells the server not to cheat and do everything asynchronously. It should *not* have a heavy performance penalty for I/O intensive writes if the client is using async writes and commits.

No, I didn't get confused ... We can use Linux as an example :) .. check out:
http://marc.info/?l=linux-nfs&m=119618886105337&w=2

-- quote

The default export might have been "async", but unless the option "sync" in /etc/exports was being ignored I was already using "sync". Nevertheless I will try to change to async and test if it makes a difference.

(one day later: )

I have now tried it and the load on the NFS server is much lower and KDE logins seem to be reasonably fast now.

-- un-quote

Quoted 2 lines

-- Wendy

I'm not sure how the problem you referred to is related to the one at hand. what I'm trying to say is that the sync _exports_ flag has stronger affect on namespace modifying workloads, like what I believe Frank's workload might be ... [snip]