What should we do about the nvme atomics mess?
Subsystems (2)
Christoph Hellwig
maintainerHi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to various problems because the limit is a mess when namespace have different logical block sizes, and it b) also causes additional issues because NVMe allows it to be different for different controllers in the same subsystem.
Commit 8695f060a029 added some sanity checks to deal with issue 2b, but we kept running into more issues with it. Partially because the check wasn't quite correct, but also because we've gotten reports of controllers that change the AWUPF value when reformatting namespaces to deal with issue 2a.
And I'm a bit lost on what to do here.
We could:
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF
I guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
not the
Keith Busch
maintainerQuoted 10 lines · Christoph Hellwig
We could:
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF
Quoted 4 lines
I guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
I like option III. The controler scoped atomic size is broken for all the reasons you mentioned, so I vote we not bother trying to make sense of it.
Quoted 11 lines · Keith Busch
Quoted 10 lines · Christoph Hellwig
We could:
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway
due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF
Quoted 9 lines
Quoted 4 lines
I guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
I like option III. The controler scoped atomic size is broken for all the reasons you mentioned, so I vote we not bother trying to make sense of it.
Agree. We might consider I. as a fixup for stable, but should continue with III going forward.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Keith Busch
maintainerQuoted 17 lines · Hannes Reinecke
Quoted 16 lines · Keith Busch
Quoted 14 lines · Christoph Hellwig
We could:
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway
due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPFI guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
Quoted 6 lines
Quoted 4 lines
I like option III. The controler scoped atomic size is broken for all the reasons you mentioned, so I vote we not bother trying to make sense of it.
Agree. We might consider I. as a fixup for stable, but should continue with III going forward.
I think the NVMe TWG might want to consider an ECN to deprecate or at least recommend against AUWPF, too.
Just to throw AWUPF a lifeline for legecy devices, we could potentially make sense of the value if Identify Controller says:
1. CMIC == 0; and
2. OACS.NMS == 0; and
3.
a. FNA.FNS == 1; or
b. NN == 1And if those conditions are true, then the controller and namespace scopes resolve to a single namespace format, so the values should be one in the same. The only way it could change, then, is a format command, which means there couldn't be an in-use filesystem depending on it not changing.
Quoted 17 lines · Keith Busch
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Keith Busch
Quoted 14 lines · Christoph Hellwig
We could:
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway
due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPFI guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I like option III. The controler scoped atomic size is broken for all the reasons you mentioned, so I vote we not bother trying to make sense of it.
Agree. We might consider I. as a fixup for stable, but should continue with III going forward.
I think the NVMe TWG might want to consider an ECN to deprecate or at least recommend against AUWPF, too.
We should really find a way to fix this in the spec, I'll be happy to add this topic and agenda so we can discuss it at a length, before that happens option III seems right way to fix it.
-ck
Christoph Hellwig
maintainerQuoted 2 lines · Keith Busch
I think the NVMe TWG might want to consider an ECN to deprecate or at least recommend against AUWPF, too.
Yeah. A wording that every controller SHOULD implement NAWUPF if it implements AWUPF might be good, eventually upgraded to a SHALL.
Quoted 5 lines
Just to throw AWUPF a lifeline for legecy devices, we could potentially make sense of the value if Identify Controller says:
1. CMIC == 0; and
2. OACS.NMS == 0; and
What is NMS meant to say? namespace management support?
Quoted 9 lines
3.
a. FNA.FNS == 1; or b. NN == 1And if those conditions are true, then the controller and namespace scopes resolve to a single namespace format, so the values should be one in the same. The only way it could change, then, is a format command, which means there couldn't be an in-use filesystem depending on it not changing.
We could. But are there many controllers where that would be the case and where people want to use atomics?
Keith Busch
maintainerQuoted 8 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
Just to throw AWUPF a lifeline for legecy devices, we could potentially make sense of the value if Identify Controller says:
1. CMIC == 0; and
2. OACS.NMS == 0; andWhat is NMS meant to say? namespace management support?
Right, namespace management support. The spec calls this field 'NMS' now.
Quoted 4 lines
Quoted 4 lines
3.
a. FNA.FNS == 1; or b. NN == 1
Quoted 8 lines
Quoted 5 lines
And if those conditions are true, then the controller and namespace scopes resolve to a single namespace format, so the values should be one in the same. The only way it could change, then, is a format command, which means there couldn't be an in-use filesystem depending on it not changing.
We could. But are there many controllers where that would be the case and where people want to use atomics?
Maybe not. I still have a lot of 1.0 compliant devices where this might apply, but I don't have a use case explicitly needing the atomic write features anyway, so it doesn't matter to me if the driver doesn't report the limits for them.
So I guess no need to work with such devices at this point, but maybe just something to consider in the unlikely event someone complains.
Quoted 6 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
Quoted 8 lines
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to
If controller-wide AWUPF is a must property, the length has to be aligned with block size.
Quoted 4 lines
various problems because the limit is a mess when namespace have different logical block sizes, and it b) also causes additional issues because NVMe allows it to be different for different controllers in the same subsystem.
The spec mentioned clearly that controller AWUPF should be supported by any namespace format:
``` Atomic Write Unit Power Fail (AWUPF): This field indicates the size of the write operation guaranteed to be written atomically to the NVM across all namespaces with any supported namespace format during a power fail or error condition. ```
So I am wondering why nvme driver can't validate NAWUN against AWUPF?
Thanks,
Ming
Keith Busch
maintainerQuoted 16 lines · Ming Lei
Quoted 6 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
Quoted 8 lines
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to
Quoted 2 lines
If controller-wide AWUPF is a must property, the length has to be aligned with block size.
What block size? The controller doesn't have one. Block sizes are properties of namespaces, not controllers or subsystems. If you have 10 namespaces with 10 different block formats, what does AUWPF mean? If the controller must report something, the only rational thing it could declare is reduced to the greatest common denominator, which is out of sync with the true value reported in the appropriately scoped NAUWPF value.
Quoted 17 lines · Keith Busch
Quoted 16 lines · Ming Lei
Quoted 14 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to
Quoted 4 lines
Quoted 2 lines
If controller-wide AWUPF is a must property, the length has to be aligned with block size.
What block size? The controller doesn't have one. Block sizes are
It should be any NS format's block size.
Quoted 6 lines
properties of namespaces, not controllers or subsystems. If you have 10 namespaces with 10 different block formats, what does AUWPF mean? If the controller must report something, the only rational thing it could declare is reduced to the greatest common denominator, which is out of sync with the true value reported in the appropriately scoped NAUWPF value.
Yes, please see the words I quoted from NVMe spec, also 6.4 Atomic Operations
mentioned: NAWUPF >= AWUPF.
Thanks,
Ming
Keith Busch
maintainerQuoted 18 lines · Ming Lei
Quoted 17 lines · Keith Busch
Quoted 16 lines · Ming Lei
Quoted 14 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If controller-wide AWUPF is a must property, the length has to be aligned with block size.
What block size? The controller doesn't have one. Block sizes are
It should be any NS format's block size.
That requires an artificial reduction to a meaningless value.
Quoted 9 lines
Quoted 6 lines
properties of namespaces, not controllers or subsystems. If you have 10 namespaces with 10 different block formats, what does AUWPF mean? If the controller must report something, the only rational thing it could declare is reduced to the greatest common denominator, which is out of sync with the true value reported in the appropriately scoped NAUWPF value.
Yes, please see the words I quoted from NVMe spec, also
6.4 Atomic Operations
mentioned:NAWUPF >= AWUPF.
The problem is when Namespace X changes its format that then alters Namesace Y's reported atomic size. That's unacceptable for any filesystem utilizing this feature.
Quoted 19 lines · Keith Busch
Quoted 18 lines · Ming Lei
Quoted 17 lines · Keith Busch
Quoted 16 lines · Ming Lei
Quoted 14 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If controller-wide AWUPF is a must property, the length has to be aligned with block size.
What block size? The controller doesn't have one. Block sizes are
It should be any NS format's block size.
That requires an artificial reduction to a meaningless value.
Any value has to be 'block size' aligned.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
properties of namespaces, not controllers or subsystems. If you have 10 namespaces with 10 different block formats, what does AUWPF mean? If the controller must report something, the only rational thing it could declare is reduced to the greatest common denominator, which is out of sync with the true value reported in the appropriately scoped NAUWPF value.
Quoted 6 lines
Quoted 2 lines
Yes, please see the words I quoted from NVMe spec, also
6.4 Atomic Operations
mentioned:NAWUPF >= AWUPF.The problem is when Namespace X changes its format that then alters Namesace Y's reported atomic size. That's unacceptable for any filesystem utilizing this feature.
When X changes its format, FS has to be umount.
The actual length(byte unit) of atomic write does not changed for Y, just the unit(block size) is changed, at least from Yi's report.
Thanks,
Ming
Quoted 29 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to various problems because the limit is a mess when namespace have different logical block sizes, and it b) also causes additional issues because NVMe allows it to be different for different controllers in the same subsystem.Commit
8695f060a029added some sanity checks to deal with issue 2b, but we kept running into more issues with it. Partially because the check wasn't quite correct, but also because we've gotten reports of controllers that change the AWUPF value when reformatting namespaces to deal with issue 2a.And I'm a bit lost on what to do here.
We could:
Quoted 7 lines
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF
I like III.
But NVMe should probably push to deprecate AUWPF, and introduce a new field that is like AUWPF but which is specified in a fixed unit, e.g. bytes or CAP.MPSMIN. (I'm thinking of e.g. Zone Append Size Limit (ZASL) which is also a per controller limit, but the value is specified in units of CAP.MPSMIN, just like the Maximum Data Transfer Size (MDTS).)
Kind regards,
Niklas
Christoph Hellwig
maintainerQuoted 5 lines · Niklas Cassel
But NVMe should probably push to deprecate AUWPF, and introduce a new field that is like AUWPF but which is specified in a fixed unit, e.g. bytes or CAP.MPSMIN. (I'm thinking of e.g. Zone Append Size Limit (ZASL) which is also a per controller limit, but the value is specified in units of CAP.MPSMIN, just like the Maximum Data Transfer Size (MDTS).)
There's not advantage in having yet another field vs mandating NAWUPF.
Quoted 29 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to various problems because the limit is a mess when namespace have different logical block sizes, and it b) also causes additional issues because NVMe allows it to be different for different controllers in the same subsystem.Commit
8695f060a029added some sanity checks to deal with issue 2b, but we kept running into more issues with it. Partially because the check wasn't quite correct, but also because we've gotten reports of controllers that change the AWUPF value when reformatting namespaces to deal with issue 2a.And I'm a bit lost on what to do here.
We could:
Quoted 7 lines
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway
due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF
This would help avoid the ambiguity in whether NABSPF is valid if nsfeat bit 1 is unset.
However, it would be nice to have an idea of how many/percentage of products it would affect today. FWIW, I only have 1x SSD which supports atomics, and it does set that bit.
I suppose we could quirk known "good" HW which relies on AWUPF (to enable atomics), but that is very far from a nice approach.
Quoted 6 lines
I guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
not the
Quoted 37 lines · Christoph Hellwig
Hi all,
I'm a bit lost on what to do about the sad state of NVMe atomic writes.
As a short reminder the main issues are:
1) there is no flag on a command to request atomic (aka non-torn) behavior, instead writes adhering to the atomicy requirements will never be torn, and writes not adhering them can be torn any time. This differs from SCSI where atomic writes have to be be explicitly requested and fail when they can't be satisfied
2) the original way to indicate the main atomicy limit is the AWUPF field, which is in Identify Controller, but specified in logical blocks which only exist at a namespace layer. This a) lead to various problems because the limit is a mess when namespace have different logical block sizes, and it b) also causes additional issues because NVMe allows it to be different for different controllers in the same subsystem.Commit
8695f060a029added some sanity checks to deal with issue 2b, but we kept running into more issues with it. Partially because the check wasn't quite correct, but also because we've gotten reports of controllers that change the AWUPF value when reformatting namespaces to deal with issue 2a.And I'm a bit lost on what to do here.
We could:
I. revert the check and the subsequent fixup. If you really want
to use the nvme atomics you already better pray a lot anyway due to issue 1)
II. limit the check to multi-controller subsystems
III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF
Quoted 6 lines
I guess for 6.16 we are limited to I. to bring us back to the previous state, but I have a really bad gut feeling about it given the really bad spec language and a lot of low quality NVMe implementations we're seeing these days.
not the
I believe there are multi-controller NVMe disks in the field (including the one I have) that do not exhibit such inconsistencies, i.e., they report a consistent AWUPF value across controllers and do not change it based on namespace format. The NVMe specification states this (quoting it from NVM-Command-Set-Specification-1.0e):
"The values (referencing AWUPF / AWUN) reported in the Identify Controller data structure are valid across all namespaces with any supported namespace format, forming a baseline value that is guaranteed not to change."
While the spec doesn’t explicitly require that AWUPF be consistent across controllers within the same subsystem, it seems to be implied. That said, I agree this should have been stated explicitly in the specification.
If vendors strictly adhered to the current spec, we likely wouldn’t be facing this issue. That said, given the current behavior, I also support approach III. However, choosing this approach effectively penalizes vendors who have implemented atomic write support correctly—that is, those who use AWUPF to advertise atomic write capabilities, do not rely on NAWUPF, and report a consistent AWUPF across controllers.
In my opinion, the proper long-term fix is to escalate this to the NVMe Technical Work Group (TWG) and propose a specification update that:
- Deprecates the use of AWUPF for advertising atomic write capabilities
- Mandates the use of NAWUPF instead
Once such a spec update is ratified, we can move forward with approach III.
Thanks,
--Nilay
Keith Busch
maintainerQuoted 9 lines · Nilay Shroff
I believe there are multi-controller NVMe disks in the field (including the one I have) that do not exhibit such inconsistencies, i.e., they report a consistent AWUPF value across controllers and do not change it based on namespace format. The NVMe specification states this (quoting it from NVM-Command-Set-Specification-1.0e):
"The values (referencing AWUPF / AWUN) reported in the Identify Controller data structure are valid across all namespaces with any supported namespace format, forming a baseline value that is guaranteed not to change."
I don't think that's a backward compatible requirement. Controllers often rescale these after a format command, and it was the only way for 1.0 and 1.1 controllers to report atomic sizes.
Lets say the controller can do 128k byte atomic writes, If all namespaces used 512b LBA format, then AWUPF would be 255. If you change one namespace format to 4k, AWUPF scales down to 31, yielding a sub-optimal result for all the other namespaces.
Quoted 3 lines
While the spec doesn´t explicitly require that AWUPF be consistent across controllers within the same subsystem, it seems to be implied. That said, I agree this should have been stated explicitly in the specification.
Considering multi-controller subsystems, some controllers might have namespaces with only 512b formats attached, and other controllers might have some 4k mixed in, so then they can't all consistently report the desired AWUPF value. They'd have to just scale AWUPF based on the largest sector size supported. Which I guess is what the current wording is guiding toward, but that just suggests host drivers disregard the value and use NAWUPF instead. So still option III.
Quoted 11 lines · Keith Busch
Quoted 9 lines · Nilay Shroff
I believe there are multi-controller NVMe disks in the field (including the one I have) that do not exhibit such inconsistencies, i.e., they report a consistent AWUPF value across controllers and do not change it based on namespace format. The NVMe specification states this (quoting it from NVM-Command-Set-Specification-1.0e):
"The values (referencing AWUPF / AWUN) reported in the Identify Controller data structure are valid across all namespaces with any supported namespace format, forming a baseline value that is guaranteed not to change."
Quoted 9 lines
I don't think that's a backward compatible requirement. Controllers often rescale these after a format command, and it was the only way for 1.0 and 1.1 controllers to report atomic sizes.
Lets say the controller can do 128k byte atomic writes, If all namespaces used 512b LBA format, then AWUPF would be 255. If you change one namespace format to 4k, AWUPF scales down to 31, yielding a sub-optimal result for all the other namespaces.
On the multi-controller disk I’ve been testing, each controller consistently reports an AWUPF value of 63. I created shared namespaces with mixed LBA formats — some using 512-byte LBAs and others using 4KB LBAs — and observed that the AWUPF value remained constant at 63 across all controllers and formats.
This implies that:
- A namespace with 4KB LBA format can support up to 256KB of atomic writes (4KB × 64),
- A namespace with 512-byte LBA format can only support up to 32KB of atomic writes (512B × 64).
So in this case, it's actually the opposite of what one might assume:
Users of namespaces with 4KB LBA format would see the best possible atomic write performance, while those using 512-byte LBA format may observe sub-optimal performance, since the maximum atomic write size scales down with smaller LBAs.
Quoted 11 lines
Quoted 3 lines
While the spec doesn´t explicitly require that AWUPF be consistent across controllers within the same subsystem, it seems to be implied. That said, I agree this should have been stated explicitly in the specification.
Considering multi-controller subsystems, some controllers might have namespaces with only 512b formats attached, and other controllers might have some 4k mixed in, so then they can't all consistently report the desired AWUPF value. They'd have to just scale AWUPF based on the largest sector size supported. Which I guess is what the current wording is guiding toward, but that just suggests host drivers disregard the value and use NAWUPF instead. So still option III.
Yes, I agree — option III seems to be the best possible way forward. However, does this mean we would disregard atomic write support for any multi-controller NVMe vendor that consistently reports a valid AWUPF value across all controllers and namespace formats, but sets NAWUPF to zero?
Thanks,
--Nilay
Christoph Hellwig
maintainerQuoted 4 lines · Nilay Shroff
So in this case, it's actually the opposite of what one might assume:
Users of namespaces with 4KB LBA format would see the best possible atomic write performance, while those using 512-byte LBA format may observe sub-optimal performance, since the maximum atomic write size scales down with smaller LBAs.
The problem is that we need to deal with the worst case and not the best case. And NVMe royally messed up there.
Quoted 11 lines · Nilay Shroff
Quoted 7 lines
Considering multi-controller subsystems, some controllers might have namespaces with only 512b formats attached, and other controllers might have some 4k mixed in, so then they can't all consistently report the desired AWUPF value. They'd have to just scale AWUPF based on the largest sector size supported. Which I guess is what the current wording is guiding toward, but that just suggests host drivers disregard the value and use NAWUPF instead. So still option III.
Yes, I agree — option III seems to be the best possible way forward. However, does this mean we would disregard atomic write support for any multi-controller NVMe vendor that consistently reports a valid AWUPF value across all controllers and namespace formats, but sets NAWUPF to zero?
Hi Nilay,
Does the drive which you are using report NAWUPF as zero (as hinted)?
If so, have you tried the following https://lore.kernel.org/linux-nvme/[email protected]/
We were considering changing the NVMe driver to not use AWUPF at all...
Thanks,
John
Quoted 13 lines · John Garry
Quoted 11 lines · Nilay Shroff
Quoted 7 lines
Considering multi-controller subsystems, some controllers might have namespaces with only 512b formats attached, and other controllers might have some 4k mixed in, so then they can't all consistently report the desired AWUPF value. They'd have to just scale AWUPF based on the largest sector size supported. Which I guess is what the current wording is guiding toward, but that just suggests host drivers disregard the value and use NAWUPF instead. So still option III.
Yes, I agree — option III seems to be the best possible way forward. However, does this mean we would disregard atomic write support for any multi-controller NVMe vendor that consistently reports a valid AWUPF value across all controllers and namespace formats, but sets NAWUPF to zero?
Quoted 7 lines
Hi Nilay,
Does the drive which you are using report NAWUPF as zero (as hinted)?
If so, have you tried the following https://lore.kernel.org/linux-nvme/[email protected]/
We were considering changing the NVMe driver to not use AWUPF at all...
Yes, I just tested your patch with the latest upstream kernel on my drive, which reports a non-zero AWUPF but a zero NAWUPF.
And your opt-in change works well on my disk. So we may consider merging this change if everyone agrees.
Thanks,
--Nilay
On 21/10/2025 16:02, Nilay Shroff wrote:
-
Quoted 7 lines
Quoted 5 lines
Does the drive which you are using report NAWUPF as zero (as hinted)?
If so, have you tried the followinghttps://lore.kernel.org/linux-nvme/20250820150220.1923826-1- [email protected]/
We were considering changing the NVMe driver to not use AWUPF at all...
Yes, I just tested your patch with the latest upstream kernel on my drive, which reports a non-zero AWUPF but a zero NAWUPF.
Do you think that you could check with the OEM for updated firmware (that reports NAWUPF)?
You IBM guys (I guess that you work together) have been a great help for the atomic writes work, and I think that it would be unfortunate to lose the atomic write capability for the drives which you use. Or lose automatic capability (like in this patch).
Thanks,
John
Quoted 6 lines · John Garry
On 21/10/2025 16:02, Nilay Shroff wrote:
-
Quoted 2 lines
Quoted 2 lines
Does the drive which you are using report NAWUPF as zero (as hinted)?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
If so, have you tried the followinghttps://lore.kernel.org/linux-nvme/20250820150220.1923826-1- [email protected]/
We were considering changing the NVMe driver to not use AWUPF at all...
Yes, I just tested your patch with the latest upstream kernel on my drive, which reports a non-zero AWUPF but a zero NAWUPF.
Do you think that you could check with the OEM for updated firmware (that reports NAWUPF)?
Yes, certainly — we can check with the disk vendor for updated firmware that correctly reports NAWUPF. I’ve already discussed this with my manager, and he’ll arrange a call with the vendor so that I can directly explain the issue and what we want from the kernel perspective. I’ll follow up with you once I’ve spoken to the vendor.
Thanks,
--Nilay
Quoted 12 lines · Nilay Shroff
Quoted 9 lines · John Garry
On 21/10/2025 16:02, Nilay Shroff wrote:
-
Quoted 5 lines
Quoted 5 lines
Does the drive which you are using report NAWUPF as zero (as hinted)?
If so, have you tried the followinghttps://lore.kernel.org/linux-nvme/20250820150220.1923826-1- [email protected]/
We were considering changing the NVMe driver to not use AWUPF at all...
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Yes, I just tested your patch with the latest upstream kernel on my drive, which reports a non-zero AWUPF but a zero NAWUPF.
Do you think that you could check with the OEM for updated firmware (that reports NAWUPF)?
Yes, certainly — we can check with the disk vendor for updated firmware that correctly reports NAWUPF. I’ve already discussed this with my manager, and he’ll arrange a call with the vendor so that I can directly explain the issue and what we want from the kernel perspective. I’ll follow up with you once I’ve spoken to the vendor.
Just an update...
We met with the vendor team and explained the current situation regarding atomic writes and the NVMe Linux kernel driver. I’m happy to report that they acknowledged the issue and agreed to provide updated firmware that advertises atomic write support using NAWUPF instead of AWUPF.
We shall first receive a test firmware build with this change. Once our test team verifies that it behaves as expected, the vendor will proceed with a formal firmware release. I’ll keep you posted once we receive the updated firmware.
Thanks,
--Nilay
Quoted 8 lines · Nilay Shroff
Quoted 6 lines
Quoted 1 line
Do you think that you could check with the OEM for updated firmware (that reports NAWUPF)?
Yes, certainly — we can check with the disk vendor for updated firmware that correctly reports NAWUPF. I’ve already discussed this with my manager, and he’ll arrange a call with the vendor so that I can directly explain the issue and what we want from the kernel perspective. I’ll follow up with you once I’ve spoken to the vendor.
Just an update...
Quoted 8 lines
We met with the vendor team and explained the current situation regarding atomic writes and the NVMe Linux kernel driver. I’m happy to report that they acknowledged the issue and agreed to provide updated firmware that advertises atomic write support using NAWUPF instead of AWUPF.
We shall first receive a test firmware build with this change. Once our test team verifies that it behaves as expected, the vendor will proceed with a formal firmware release. I’ll keep you posted once we receive the updated firmware.
Thanks for the update. It's good to know that this vendor will provide the required FW update.
Cheers
Quoted 14 lines · John Garry
Quoted 13 lines · Nilay Shroff
Quoted 6 lines
Quoted 1 line
Do you think that you could check with the OEM for updated firmware (that reports NAWUPF)?
Yes, certainly — we can check with the disk vendor for updated firmware that correctly reports NAWUPF. I’ve already discussed this with my manager, and he’ll arrange a call with the vendor so that I can directly explain the issue and what we want from the kernel perspective. I’ll follow up with you once I’ve spoken to the vendor.
Just an update...
We met with the vendor team and explained the current situation regarding atomic writes and the NVMe Linux kernel driver. I’m happy to report that they acknowledged the issue and agreed to provide updated firmware that advertises atomic write support using NAWUPF instead of AWUPF.
Quoted 6 lines
Quoted 3 lines
We shall first receive a test firmware build with this change. Once our test team verifies that it behaves as expected, the vendor will proceed with a formal firmware release. I’ll keep you posted once we receive the updated firmware.
Thanks for the update. It's good to know that this vendor will provide the required FW update.
An update...
IBM has now formally received a firmware update that adheres to the (proposed) Linux NVMe driver requirements for atomic write support. With this updated firmware, the controller now reports AWUPF = 0 in the Identify Controller data structure. All namespace-specific atomic write limits, including NAWUPF, are reported exclusively via the Identify Namespace data structure.
Additionally, the atomic write limits are now correctly scaled based on the LBA format, as expected. The results below illustrate this behavior:
$ nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme5n2 /dev/ng5n2 3D60A04906N1 1.6TB NVMe Gen4 U.2 SSD IV 0x1 12.44 GB / 204.84 GB 4 KiB + 0 B REV.BAS8 /dev/nvme5n3 /dev/ng5n3 3D60A04906N1 1.6TB NVMe Gen4 U.2 SSD IV 0x3 0.00 B / 25.61 GB 512 B + 0 B REV.BAS8
$ nvme id-ctrl /dev/nvme5 | grep awupf awupf : 0
$ nvme id-ns /dev/nvme5n2 | grep -A4 nawupf nawupf : 63 nacwu : 63 nabsn : 0 nabo : 0 nabspf : 0
nvme id-ns /dev/nvme5n3 | grep -A4 nawupf
nawupf : 511 ==> As expected, the atomic write limit scales naturally for 512B LBA
nacwu : 511
nabsn : 0
nabo : 0
nabspf : 0
With this update, atomic write support is now clearly and unambiguously expressed at the namespace level, while the controller-level AWUPF is set to zero, avoiding ambiguity and aligning with the Linux NVMe driver expectations. Based on this outcome, we could consider:
- Recommending other vendors adopt the same reporting model, and
- As it was already proposed earlier, updating the Linux NVMe kernel driver to disregard atomic write support for devices that report a non-zero AWUPF at the controller level.
Thanks,
--Nilay
Quoted 35 lines · Nilay Shroff
Quoted 2 lines
Thanks for the update. It's good to know that this vendor will provide the required FW update.
An update...
IBM has now formally received a firmware update that adheres to the (proposed) Linux NVMe driver requirements for atomic write support. With this updated firmware, the controller now reports AWUPF = 0 in the Identify Controller data structure. All namespace-specific atomic write limits, including NAWUPF, are reported exclusively via the Identify Namespace data structure.
Additionally, the atomic write limits are now correctly scaled based on the LBA format, as expected. The results below illustrate this behavior:$ nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme5n2 /dev/ng5n2 3D60A04906N1 1.6TB NVMe Gen4 U.2 SSD IV 0x1 12.44 GB / 204.84 GB 4 KiB + 0 B REV.BAS8 /dev/nvme5n3 /dev/ng5n3 3D60A04906N1 1.6TB NVMe Gen4 U.2 SSD IV 0x3 0.00 B / 25.61 GB 512 B + 0 B REV.BAS8$ nvme id-ctrl /dev/nvme5 | grep awupf awupf : 0$ nvme id-ns /dev/nvme5n2 | grep -A4 nawupf nawupf : 63 nacwu : 63 nabsn : 0 nabo : 0 nabspf : 0nvme id-ns
/dev/nvme5n3| grep -A4 nawupf
nawupf : 511 ==> As expected, the atomic write limit scales naturally for 512B LBA
nacwu : 511
nabsn : 0
nabo : 0
nabspf : 0
Quoted 3 lines
With this update, atomic write support is now clearly and unambiguously expressed at the namespace level, while the controller-level AWUPF is set to zero, avoiding ambiguity and aligning with the Linux NVMe driver expectations.
Thanks for the info. And it's good to hear about this FW update.
Quoted 6 lines
Based on this
outcome, we could consider:- Recommending other vendors adopt the same reporting model, and
- As it was already proposed earlier, updating the Linux NVMe kernel driver to disregard atomic write support for devices that report a non-zero AWUPF at the controller level.
Yeah, we can just stop using AWUPF always. Or can further consider my proposed change to allow AWUPF to be used based on an opt-in, below.
Keith, Christoph, Any further thoughts on this?
https://lore.kernel.org/linux-nvme/[email protected]/
Cheers,
John
Christoph Hellwig
maintainerQuoted 10 lines · John Garry
Quoted 3 lines
- Recommending other vendors adopt the same reporting model, and
- As it was already proposed earlier, updating the Linux NVMe kernel driver to disregard atomic write support for devices that report a non-zero AWUPF at the controller level.Yeah, we can just stop using AWUPF always. Or can further consider my proposed change to allow AWUPF to be used based on an opt-in, below.
Keith, Christoph, Any further thoughts on this?
I'd love to drop support for the controller-level AWUPF.
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines
Quoted 1 line
o AWUPF at the controller level.
Yeah, we can just stop using AWUPF always. Or can further consider my proposed change to allow AWUPF to be used based on an opt-in, below.
Keith, Christoph, Any further thoughts on this?
https://lore.kernel.org/linux-nvme/20250820150220.1923826-1- [email protected]/
I'd love to drop support for the controller-level AWUPF.
Yeah, I was just going to send a patch for that.
cheers
Christoph,
Quoted 1 line
I'd love to drop support for the controller-level AWUPF.
Yup, I agree. It's utterly broken.
Signature · 1 line
Martin K. Petersen