Skip to contentSkip to conversation

Do we need an opt-in for file systems use of hw atomic writes?

DISCUSSION Christoph Hellwig 10 participants 28 messages Jul 14, 2025 – Aug 21, 2025

Subsystems (3)
lore
Share & tools
lore

Christoph Hellwig

author maintainer
#0 lore raw

Hi all,

I'm currently trying to sort out the nvme atomics limits mess, and between that, the lack of a atomic write command in nvme, and the overall degrading quality of cheap consumer nvme devices I'm starting to free really uneasy about XFS using hardware atomics by default without an explicit opt-in, as broken atomics implementations will lead to really subtle data corruption.

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

#1 lore raw
Quoted 11 lines · Christoph Hellwig

Hi all,

I'm currently trying to sort out the nvme atomics limits mess, and between that, the lack of a atomic write command in nvme, and the overall degrading quality of cheap consumer nvme devices I'm starting to free really uneasy about XFS using hardware atomics by default without an explicit opt-in, as broken atomics implementations will lead to really subtle data corruption.

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

How common do we think broken atomics implementations; is this something that we could solve using a blacklist of broken devices?

It used to be the case that broken flash devices would get bricked when trim commands would get sent racing with write requests. But over time, the broken devices died out (in some cases, when the companies selling the broken SSD's died out :-). It would have been annoying if we had to explicitly enable trim support in 2025 just because there were some broken SSD's existed a decade or more ago.

- Ted

author maintainer
#2 lore raw
Quoted 5 lines · Theodore Ts'o
Quoted 2 lines

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

How common do we think broken atomics implementations; is this something that we could solve using a blacklist of broken devices?

I don't know. But cheap consumer SSDs can basically exhibit any brokenness you can imagine. And claiming to support atomics basically just means filling out a single field in identify with a non-zero value. So my hopes of only seeing it in a few devices is low, moreover we will only notice it was broken when people lost data.

#8 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig

Hi all,

I'm currently trying to sort out the nvme atomics limits mess, and between that, the lack of a atomic write command in nvme, and the overall degrading quality of cheap consumer nvme devices I'm starting to free really uneasy about XFS using hardware atomics by default without an explicit opt-in, as broken atomics implementations will lead to really subtle data corruption.

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

But isn't this just an NVMe issue? I would assume that we would look at such an option in the NVMe driver (to opt in when we are concerned about the implementation), and not the FS. SCSI is ok AFAIK.

And we also have bdev fops supporting atomic writes, so any opt in method needs to cover that.

#9 lore raw
Quoted 10 lines · John Garry
Quoted 9 lines · Christoph Hellwig

Hi all,

I'm currently trying to sort out the nvme atomics limits mess, and between that, the lack of a atomic write command in nvme, and the overall degrading quality of cheap consumer nvme devices I'm starting to free really uneasy about XFS using hardware atomics by default without an explicit opt-in, as broken atomics implementations will lead to really subtle data corruption.

Quoted 6 lines
Quoted 2 lines

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

But isn't this just an NVMe issue? I would assume that we would look at such an option in the NVMe driver (to opt in when we are concerned about the implementation), and not the FS. SCSI is ok AFAIK.

SCSI is a better standard, and modulo USB devices doesn't have as much of an issue with cheap consumer devices.

But form the file system POV we've spent the last decade or so hardening file systems against hardware failures, so now suddenly using such a high risk feature automatically feels a bit odd.

#10 lore raw
Quoted 18 lines · Christoph Hellwig
Quoted 10 lines · John Garry
Quoted 9 lines · Christoph Hellwig

Hi all,

I'm currently trying to sort out the nvme atomics limits mess, and between that, the lack of a atomic write command in nvme, and the overall degrading quality of cheap consumer nvme devices I'm starting to free really uneasy about XFS using hardware atomics by default without an explicit opt-in, as broken atomics implementations will lead to really subtle data corruption.

Quoted 6 lines
Quoted 2 lines

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

But isn't this just an NVMe issue? I would assume that we would look at such an option in the NVMe driver (to opt in when we are concerned about the implementation), and not the FS. SCSI is ok AFAIK.

Quoted 7 lines

SCSI is a better standard, and modulo USB devices doesn't have as much of an issue with cheap consumer devices.

But form the file system POV we've spent the last decade or so hardening file systems against hardware failures, so now suddenly using such a high risk feature automatically feels a bit odd.

I see. I figure that something like a FS_XFLAG could be used for that. But we should still protect bdev fops users as well.

JFYI, I have done a good bit of HW and SW-based atomic powerfail testing with fio on a Linux dev board, so there is a decent method available for users to verify their HW atomics. But then testing power failures is not always practical. Crashing the kernel only tests AWUN, and AWUPF (for NVMe).

#3 lore raw
In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Theodore Ts'o
Quoted 2 lines

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

Quoted 8 lines
Quoted 2 lines

How common do we think broken atomics implementations; is this something that we could solve using a blacklist of broken devices?

I don't know. But cheap consumer SSDs can basically exhibit any brokenness you can imagine. And claiming to support atomics basically just means filling out a single field in identify with a non-zero value. So my hopes of only seeing it in a few devices is low, moreover we will only notice it was broken when people lost data.

Do you want to handle it the same way as we do discard-zeroes-data and have a quirks list of devices we trust? Though I can hardly talk, knowing the severe limitations of allowlists vs. product managers trying to win benchmarks with custom firmware. :(

--D

#24 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig

Hi all,

I'm currently trying to sort out the nvme atomics limits mess, and between that, the lack of a atomic write command in nvme, and the overall degrading quality of cheap consumer nvme devices I'm starting to free really uneasy about XFS using hardware atomics by default without an explicit opt-in, as broken atomics implementations will lead to really subtle data corruption.

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

This isn't a filesystem question - this is a question about what features the block device should expose by default to the user/filesystem by default.

Block device feature configuration is typically done at hotplug time with udev rules. Require the user to add a custom udev rule for the block device to enable hardware atomics if you are concerned that hardware atomic writes are problematic.

Once the user has opted in to having their bdev feature activated, then the filesystem should be able to detect and use it automatically.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw
In reply to Christoph Hellwig

Christoph,

Quoted 5 lines

I don't know. But cheap consumer SSDs can basically exhibit any brokenness you can imagine. And claiming to support atomics basically just means filling out a single field in identify with a non-zero value. So my hopes of only seeing it in a few devices is low, moreover we will only notice it was broken when people lost data.

It's very unfortunate that AWUN/AWUPF are useless. Even when it comes to something as simple as sanity checking the reported NAWUN/NAWUPF.

For PCIe transport devices maybe we could consider adding an additional heuristic based on something like PLP or VWC?

Signature · 1 line
Martin K. Petersen
author maintainer
#4 lore raw
In reply to Darrick J. Wong
Quoted 4 lines · Darrick J. Wong

Do you want to handle it the same way as we do discard-zeroes-data and have a quirks list of devices we trust? Though I can hardly talk, knowing the severe limitations of allowlists vs. product managers trying to win benchmarks with custom firmware. :(

I don't think whitelists are a good idea. I'd expect the admin to opt into it.

author maintainer
#6 lore raw
In reply to Martin K. Petersen
Quoted 2 lines · Martin K. Petersen

For PCIe transport devices maybe we could consider adding an additional heuristic based on something like PLP or VWC?

What would you check there? Atomic writes work perfectly fine if not better with volatile write caches.

author maintainer
#11 lore raw
In reply to John Garry
Quoted 2 lines · John Garry

I see. I figure that something like a FS_XFLAG could be used for that. But we should still protect bdev fops users as well.

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

Quoted 6 lines

JFYI, I have done a good bit of HW and SW-based atomic powerfail testing with fio on a Linux dev board, so there is a decent method available for users to verify their HW atomics. But then testing power failures is not always practical. Crashing the kernel only tests AWUN, and AWUPF (for NVMe).

Yes. There's some ways to emulate power fail for file system level power fail testing using dm-log-writes and similar, but that doesn't help at all with testing the power fail behavior of the device which we rely on here.

author maintainer
#25 lore raw
In reply to Dave Chinner
Quoted 3 lines · Dave Chinner

This isn't a filesystem question - this is a question about what features the block device should expose by default to the user/filesystem by default.

It's both. As I said before we've spent a lot of time making the file systems less reliant on hardware getting everything right by adding checksums (for metadata everywhere, and non-XFS file systems for data), lsn verifications, etc. And now we go all in to trust a new (in case of nvme very much misdesigned) feature.

I'm perfectly fine offering that use, but I'm a lot less excited by automatically using using it due to my deep mistrust of hardware.

#12 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · John Garry

I see. I figure that something like a FS_XFLAG could be used for that. But we should still protect bdev fops users as well.

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

For all these options, the admin must know that the atomic behaviour of their disk is as advertised - I am not sure how realistic it is.

Apart from this, it would be nice to have an idea of how to handle the NVMe. About this:

" III. don't allow atomics on controllers that only report AWUPF and
limit support to controllers that support that more sanely defined NAWUPF"

Would it be possible to also have a driver opt-in for those controllers which don't support NAWUPF?

Thanks,
John

author maintainer
#13 lore raw
Quoted 6 lines · John Garry
Quoted 3 lines

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

For all these options, the admin must know that the atomic behaviour of their disk is as advertised - I am not sure how realistic it is.

Well, who else would know it, or rather who else can do the risk calculation?

I'm not worried about Oracle cloud running data bases on drives written to their purchase spec and validated by them.

I'm worried about $RANDOMUSER running $APPLICATION here that thing atomic write APIs are nice (they finally are) and while that works fine with the software implemenetation and even reasonably high end consumer devices, they now get the $CHEAPO SSD off Alibab and while things work fine their entire browinshistory / ledger / movie data base or whatever is toast and the file system gets blamed.

#20 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · John Garry

I see. I figure that something like a FS_XFLAG could be used for that. But we should still protect bdev fops users as well.

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

If we were to make this a mount option it would be really really ugly. Either it is a filesystem specific mount option and then we have the problem that we're ending up with different mount option names per-filesystem.

And for a VFS generic mount option this is way too specific. It would be extremely misplaced if we start accumulating hardware opt-out/opt-in mount options on the VFS layer.

It feels like this is something that needs to be done on the block layer. IOW, maybe add generic block layer ioctls or a per-device sysfs entry that allows to turn atomic writes on or off. That information would then also potentially available to the filesystem to e.g., generate an info message during mount that hardware atomics are used or aren't used. Because ultimately the block layer is where the decision needs to be made.

author maintainer
#21 lore raw
Quoted 8 lines · Christian Brauner
Quoted 3 lines

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

If we were to make this a mount option it would be really really ugly. Either it is a filesystem specific mount option and then we have the problem that we're ending up with different mount option names per-filesystem.

Not that I'm arguing for a mount option (this should be sticky), but we've had plenty of fs parsed mount options with common semantics.

Quoted 7 lines

It feels like this is something that needs to be done on the block layer. IOW, maybe add generic block layer ioctls or a per-device sysfs entry that allows to turn atomic writes on or off. That information would then also potentially available to the filesystem to e.g., generate an info message during mount that hardware atomics are used or aren't used. Because ultimately the block layer is where the decision needs to be made.

The block layer just passes things through.

#23 lore raw
In reply to Christian Brauner
Quoted 8 lines · Christian Brauner

It feels like this is something that needs to be done on the block layer. IOW, maybe add generic block layer ioctls or a per-device sysfs entry that allows to turn atomic writes on or off. That information would then also potentially available to the filesystem to e.g., generate an info message during mount that hardware atomics are used or aren't used. Because ultimately the block layer is where the decision needs to be made.

I'd really like it if we can edit the atomic write granularity by writing to the sysfs file to make it easier to test the atomic write codepaths in the file system.

So I'd suggest combining this with John Garry's suggestion to allow atomic writes by default on NVMe devices that report NAWUPF, not to ignore AWUPF. If system admistrators need to make atomic writes on legacy devices that only report AWUPF, they can manually set the atomic write granulairty. And if they screw up --- well, that's on them.

And file system developers who don't care about data safety on power failure (which we can't directly test via fstests anyway), but just want to test the code paths, we can manually write to the sysfs file as well.

Cheers,

- Ted

#22 lore raw
In reply to Christoph Hellwig
Quoted 21 lines · Christoph Hellwig
Quoted 8 lines · Christian Brauner
Quoted 3 lines

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

If we were to make this a mount option it would be really really ugly. Either it is a filesystem specific mount option and then we have the problem that we're ending up with different mount option names per-filesystem.

Not that I'm arguing for a mount option (this should be sticky), but we've had plenty of fs parsed mount options with common semantics.

Quoted 7 lines

It feels like this is something that needs to be done on the block layer. IOW, maybe add generic block layer ioctls or a per-device sysfs entry that allows to turn atomic writes on or off. That information would then also potentially available to the filesystem to e.g., generate an info message during mount that hardware atomics are used or aren't used. Because ultimately the block layer is where the decision needs to be made.

Quoted 1 line

The block layer just passes things through.

We already have bdev_can_atomic_write() which checks whether the underlying device is capable of hardware assisted atomic writes. If that's the case the filesystem currently just uses them, fine.

So it is possible to implement an ioctl() that allows an administrator to mark a device as untrusted for hardware assisted atomic writes.

This is also nice is because this can be integrated with udev easily. If a device is know to have broken hardware assisted atomic writes then add the device into systemd-udev's hardware database (hwdb).

When systemd-udev sees that device show up during boot it will automatically mark that device as having broken atomic write support and any mount of that device will have the filesystem immediately see the broken hardware assisted atomic write support in bdev_can_atomic_write() and not use it.

Fwiw, this pattern is already used for other stuff. For example for the iocost stuff that udev will auto-apply if known. The broken atomic write stuff would fit very well in there. Either it's an allowlist or a denylist.

commit 6b8e90545e918a4653281b3672a873e948f12b65

Author:     Gustavo Noronha Silva <[email protected]>
AuthorDate: Mon May 2 14:02:23 2022 -0300
Commit:     Lennart Poettering <[email protected]>
CommitDate: Thu Apr 20 16:45:57 2023 +0200

Apply known iocost solutions to block devices

Meta's resource control demo project[0] includes a benchmark tool that can be used to calculate the best iocost solutions for a given SSD.

[0]: https://github.com/facebookexperimental/resctl-demo

A project[1] has now been started to create a publicly available database of results that can be used to apply them automatically.

[1]: https://github.com/iocost-benchmark/iocost-benchmarks

This change adds a new tool that gets triggered by a udev rule for any block device and queries the hwdb for known solutions. The format for the hwdb file that is currently generated by the github action looks like this:

# This file was auto-generated on Tue, 23 Aug 2022 13:03:57 +0000.
# From the following commit:
# https://github.com/iocost-benchmark/iocost-benchmarks/commit/ca82acfe93c40f21d3b513c055779f43f1126f88
#
# Match key format:
# block:<devpath>:name:<model name>:

# 12 points, MOF=[1.346,1.346], aMOF=[1.249,1.249]
block:*:name:HFS256GD9TNG-62A0A:fwver:*:
IOCOST_SOLUTIONS=isolation isolated-bandwidth bandwidth naive
IOCOST_MODEL_ISOLATION=rbps=1091439492 rseqiops=52286 rrandiops=63784 wbps=192329466 wseqiops=12309 wrandiops=16119 IOCOST_QOS_ISOLATION=rpct=0.00 rlat=8807 wpct=0.00 wlat=59023 min=100.00 max=100.00 IOCOST_MODEL_ISOLATED_BANDWIDTH=rbps=1091439492 rseqiops=52286 rrandiops=63784 wbps=192329466 wseqiops=12309 wrandiops=16119 IOCOST_QOS_ISOLATED_BANDWIDTH=rpct=0.00 rlat=8807 wpct=0.00 wlat=59023 min=100.00 max=100.00 IOCOST_MODEL_BANDWIDTH=rbps=1091439492 rseqiops=52286 rrandiops=63784 wbps=192329466 wseqiops=12309 wrandiops=16119 IOCOST_QOS_BANDWIDTH=rpct=0.00 rlat=8807 wpct=0.00 wlat=59023 min=100.00 max=100.00 IOCOST_MODEL_NAIVE=rbps=1091439492 rseqiops=52286 rrandiops=63784 wbps=192329466 wseqiops=12309 wrandiops=16119 IOCOST_QOS_NAIVE=rpct=99.00 rlat=8807 wpct=99.00 wlat=59023 min=75.00 max=100.00

The IOCOST_SOLUTIONS key lists the solutions available for that device in the preferred order for higher isolation, which is a reasonable default for most client systems. This can be overriden to choose better defaults for custom use cases, like the various data center workloads.

The tool can also be used to query the known solutions for a specific device or to apply a non-default solution (say, isolation or bandwidth).

#7 lore raw
In reply to Christoph Hellwig

Hi Christoph!

Quoted 5 lines
Quoted 2 lines

For PCIe transport devices maybe we could consider adding an additional heuristic based on something like PLP or VWC?

What would you check there? Atomic writes work perfectly fine if not better with volatile write caches.

What I propose is making sure we only enable atomics when several independent device-reported values line up and are mutually consistent. Just like we do in SCSI.

Maybe something like this:

if (NSFEAT & NSABP &&

      is_power_of_2(NAWUPF) &&
      NAWUPF <= NAWUN &&
      NAWUPF <= MDTS &&
      NAWUPF % NPWG == 0)

It would be good to have more Identify Controller data in there to validate against. But since we want to fix up AWUN and AWUPF in the spec, we shouldn't depend on those.

I just wonder if there is something else from either PCIe config space or Identify Controller we could add to weed out rando consumer devices that seed their Identify buffers with garbage. A heuristic like "you wouldn't possibly want to enable atomics unless the drive also supported feature XYZ"...

Signature · 1 line
Martin K. Petersen
maintainer
#26 lore raw
In reply to Christoph Hellwig
Quoted 2 lines · Christoph Hellwig

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

IMO, if the block device's limits reports atomic capabilities, it's fair game for any in kernel use. These are used outside of filesystems too, like through raw block fops.

We've already settled on discarding problematic nvme attributes from consideration. Is there something beyond that you've really found? If so, maybe we should continue down the path of splitting more queue limits into "hardware" and "user" values, and make filesystems subscribe to the udev value where it defaults to "unsupported" for untrusted devices.

#27 lore raw
Quoted 8 lines · Keith Busch
Quoted 2 lines · Christoph Hellwig

Is is just me, or would it be a good idea to require an explicit opt-in to user hardware atomics?

IMO, if the block device's limits reports atomic capabilities, it's fair game for any in kernel use. These are used outside of filesystems too, like through raw block fops.

Quoted 7 lines

We've already settled on discarding problematic nvme attributes from consideration. Is there something beyond that you've really found? If so, maybe we should continue down the path of splitting more queue limits into "hardware" and "user" values, and make filesystems subscribe to the udev value where it defaults to "unsupported" for untrusted devices.

If we're going down the path of disregarding atomic write support for NVMe devices that don't report NAWUPF, then we likely need an opt-in mechanism for users who trust their device to have a sane AWUPF value.

For example, consider an NVMe disk that does not support NAWUPF, but does consistently support AWUPF across all namespaces and for different LBA sizes. In such cases, I would still want to enable atomic writes on this disk, even if the kernel driver marks it as "unsupported" due to missing NAWUPF.

Having an explicit user opt-in mechanism in such scenarios would be very useful, allowing advanced users to take advantage of hardware capabilities they trust, despite conservative kernel defaults.

Thanks,
--Nilay

#14 lore raw
In reply to Christoph Hellwig
Quoted 14 lines · Christoph Hellwig
Quoted 6 lines · John Garry
Quoted 3 lines

I'm not sure a XFLAG is all that useful. It's not really a per-file persistent thing. It's more of a mount option, or better persistent mount-option attr like we did for autofsck.

For all these options, the admin must know that the atomic behaviour of their disk is as advertised - I am not sure how realistic it is.

Well, who else would know it, or rather who else can do the risk calculation?

I'm not worried about Oracle cloud running data bases on drives written to their purchase spec and validated by them.

Quoted 8 lines

I'm worried about $RANDOMUSER running $APPLICATION here that thing atomic write APIs are nice (they finally are) and while that works fine with the software implemenetation and even reasonably high end consumer devices, they now get the $CHEAPO SSD off Alibab and while things work fine their entire browinshistory / ledger / movie data base or whatever is toast and the file system gets blamed.

Hi Christoph,

nothing has been happening on this thread for a while. I figure that it is because we have no good or obvious options.

I think that it's better deal with the NVMe driver handling of AWUPF first, as this applies to block fops as well.

As for the suggestion to have an opt-in to use AWUPF, you wrote above that users may not know when to enable this opt-in or not.

It seems to me that we can give the option, but clearly label that it is potentially dangerous. Hopefully the $RANDOMUSER with the $CHEAPO SSD will be wise and steer clear.

If we always ignore AWUPF, I fear that lots of sound NVMe implementations will be excluded from HW atomics.

Thanks,
John

author maintainer
#15 lore raw
Quoted 9 lines · John Garry

nothing has been happening on this thread for a while. I figure that it is because we have no good or obvious options.

I think that it's better deal with the NVMe driver handling of AWUPF first, as this applies to block fops as well.

As for the suggestion to have an opt-in to use AWUPF, you wrote above that users may not know when to enable this opt-in or not.

Quoted 6 lines

It seems to me that we can give the option, but clearly label that it is potentially dangerous. Hopefully the $RANDOMUSER with the $CHEAPO SSD will be wise and steer clear.

If we always ignore AWUPF, I fear that lots of sound NVMe implementations will be excluded from HW atomics.

I think ignoring AWUPF is a good idea, but I've also hard some folks not liking that.

The reason why I prefer a mount option is because we add that to fstab and the kernel command line easily. For block layer or driver options we'd either need a sysfs file which is always annoying to apply at boot time, or a module option which has the downside of applying to all devices.

#16 lore raw
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · John Garry

nothing has been happening on this thread for a while. I figure that it is because we have no good or obvious options.

I think that it's better deal with the NVMe driver handling of AWUPF first, as this applies to block fops as well.

As for the suggestion to have an opt-in to use AWUPF, you wrote above that users may not know when to enable this opt-in or not.

It seems to me that we can give the option, but clearly label that it is potentially dangerous. Hopefully the $RANDOMUSER with the $CHEAPO SSD will be wise and steer clear.

Quoted 5 lines
Quoted 2 lines

If we always ignore AWUPF, I fear that lots of sound NVMe implementations will be excluded from HW atomics.

I think ignoring AWUPF is a good idea, but I've also hard some folks not liking that.

Disabling reading AWUPF would be the best way to know that for sure :)

Quoted 5 lines

The reason why I prefer a mount option is because we add that to fstab and the kernel command line easily. For block layer or driver options we'd either need a sysfs file which is always annoying to apply at boot time,

Could system-udev auto enable for us via sysfs file or ioctl?

Quoted 2 lines

or a module option which has the downside of applying to all devices.

About the mount option, I suppose that it won't do much harm - it's just a bit of extra work to configure.

I just fear that admins will miss enabling it or not enable it out of doubt and users won't see the benefit of HW atomics.

#17 lore raw
Quoted 15 lines · John Garry
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · John Garry

nothing has been happening on this thread for a while. I figure that it is because we have no good or obvious options.

I think that it's better deal with the NVMe driver handling of AWUPF first, as this applies to block fops as well.

As for the suggestion to have an opt-in to use AWUPF, you wrote above that users may not know when to enable this opt-in or not.

It seems to me that we can give the option, but clearly label that it is potentially dangerous. Hopefully the $RANDOMUSER with the $CHEAPO SSD will be wise and steer clear.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

If we always ignore AWUPF, I fear that lots of sound NVMe implementations will be excluded from HW atomics.

I think ignoring AWUPF is a good idea, but I've also hard some folks not liking that.

Disabling reading AWUPF would be the best way to know that for sure :)

What is the likelihood of convincing the nvme standards folks to add a new command for write-untorn that doesn't just silently fail if you get the parameters wrong?

Quoted 4 lines
Quoted 4 lines

The reason why I prefer a mount option is because we add that to fstab and the kernel command line easily. For block layer or driver options we'd either need a sysfs file which is always annoying to apply at boot time,

(Yuck, mount options, look how poorly that went for dax= ;))

Quoted 1 line

Could system-udev auto enable for us via sysfs file or ioctl?

Userspace controllable sysfs configuration knobs like discard_max_bytes and discard_max_hw_bytes work well with that model. The nvme layer can set atomic_write_bytes to zero by default, and a udev rule can change it up to atomic_write_max_hw_bytes.

That's not /so/ bad if you can either get the udev rulefile merged into systemd, or dropped in via clod-init or something.

--D

Quoted 11 lines
Quoted 2 lines

or a module option which has the downside of applying to all devices.

About the mount option, I suppose that it won't do much harm - it's just a bit of extra work to configure.

I just fear that admins will miss enabling it or not enable it out of doubt and users won't see the benefit of HW atomics.

author maintainer
#18 lore raw
Quoted 3 lines · Darrick J. Wong

What is the likelihood of convincing the nvme standards folks to add a new command for write-untorn that doesn't just silently fail if you get the parameters wrong?

In my experience that depends mostly on how much a big customers for NVMe hardware is asking for it. Hint: while Oracle isn't in the absolute top tier of the influence scale it probably is just below.

maintainer
#19 lore raw
In reply to John Garry
Quoted 3 lines · John Garry

If we always ignore AWUPF, I fear that lots of sound NVMe implementations will be excluded from HW atomics.

It's not that they're excluded from HW atomics. They're just excluded from the block layer's attributes and guard rails. People were using NVMe atomics long before those block layer features, at least.