Skip to contentSkip to conversation

Write atomicity guarantees

DISCUSSION Matthew Wilcox 4 participants 7 messages Apr 24, 2014 1 message not in the archive VFS

lore
Share & tools
lore
#0 lore raw

NVMe allows the drive to tell the host what atomicity guarantees it provides for a write command. At the moment, I don't think Linux has a way for the driver to pass that information up to the filesystem.

The value that is most interesting to report is Atomic Write Unit Power Fail ("if you send a write no larger than this, the drive guarantees to write all of it or none of it"), minimum value 1 sector. [1]

There's a proposal before the NVMe workgroup to add a boundary size/offset to modify AWUPF ("except if you cross this boundary, then AWUPF is not guaranteed"). Think RAID stripe crossing.

So, three questions. Is there somewhere already to pass boundary information up to the filesystem? Can filesystems make use of a larger atomic write unit than a single sector? And, if the device is internally a RAID device, is knowing the boundary size/offset useful?

[1] There is also Atomic Write Unit Normal ("if you send two writes, neither of which is larger than this, subsequent reads will get either one or the other, not a mixture of both"), which I don't think we care about because the page cache prevents us from sending two writes which overlap with each other.

#1 lore raw
Quoted 19 lines · Matthew Wilcox

NVMe allows the drive to tell the host what atomicity guarantees it provides for a write command. At the moment, I don't think Linux has a way for the driver to pass that information up to the filesystem.

The value that is most interesting to report is Atomic Write Unit Power Fail ("if you send a write no larger than this, the drive guarantees to write all of it or none of it"), minimum value 1 sector. [1]

There's a proposal before the NVMe workgroup to add a boundary size/offset to modify AWUPF ("except if you cross this boundary, then AWUPF is not guaranteed"). Think RAID stripe crossing.

So, three questions. Is there somewhere already to pass boundary information up to the filesystem? Can filesystems make use of a larger atomic write unit than a single sector? And, if the device is internally a RAID device, is knowing the boundary size/offset useful?

Quoted 5 lines

[1] There is also Atomic Write Unit Normal ("if you send two writes, neither of which is larger than this, subsequent reads will get either one or the other, not a mixture of both"), which I don't think we care about because the page cache prevents us from sending two writes which overlap with each other.

I think we really need the atomics to be vectored. Send N writes which as a unit are not larger than X, but which may span anywhere on device.
An array with writeback cache, or a log structured squirrel in the FTL
should be able to provide this pretty easily?

The immediate use case is mysql (16K writes) on a fragmented filesystem.
The FS needs to be able to collect a single atomic write made up of N
4K sectors.

-chris

#2 lore raw
Quoted 28 lines · Chris Mason
Quoted 20 lines · Matthew Wilcox

NVMe allows the drive to tell the host what atomicity guarantees it provides for a write command. At the moment, I don't think Linux has a way for the driver to pass that information up to the filesystem.

The value that is most interesting to report is Atomic Write Unit Power Fail ("if you send a write no larger than this, the drive guarantees to write all of it or none of it"), minimum value 1 sector. [1]

There's a proposal before the NVMe workgroup to add a boundary size/offset to modify AWUPF ("except if you cross this boundary, then AWUPF is not guaranteed"). Think RAID stripe crossing.

So, three questions. Is there somewhere already to pass boundary information up to the filesystem? Can filesystems make use of a larger atomic write unit than a single sector? And, if the device is internally a RAID device, is knowing the boundary size/offset useful?

Quoted 5 lines

[1] There is also Atomic Write Unit Normal ("if you send two writes, neither of which is larger than this, subsequent reads will get either one or the other, not a mixture of both"), which I don't think we care about because the page cache prevents us from sending two writes which overlap with each other.

Quoted 8 lines

I think we really need the atomics to be vectored. Send N writes which as a unit are not larger than X, but which may span anywhere on device. An array with writeback cache, or a log structured squirrel in the FTL should be able to provide this pretty easily?

The immediate use case is mysql (16K writes) on a fragmented filesystem. The FS needs to be able to collect a single atomic write made up of N 4K sectors.

How big does N need to be before it starts to be generally useful? Here it seems we're talking on the order to tens of writes, but for the upper bound Dave said that N could be in the hundreds of thousands [1].

--
Dan

[1]: http://marc.info/?l=linux-fsdevel&m=139262740324307&w=2

#6 lore raw
In reply to a message not in the archive
Quoted 2 lines · Purush Gupta

That would require a new NVMe write command semantic with descriptors currently its not there.

Yes ... I don't want to start defining such a command on linux-fsdevel. What I'm hearing is that there's no benefit to a device that can guarantee to write multiple contiguous sectors in a non-torn manner over a device that can write a single sector in a non-torn manner.

For any real benefit, filesystems need (and Linux needs to introduce plumbing for) vectored atomic writes.

#7 lore raw
Quoted 4 lines · Matthew Wilcox
Quoted 2 lines · Purush Gupta

That would require a new NVMe write command semantic with descriptors currently its not there.

Quoted 8 lines

Yes ... I don't want to start defining such a command on linux-fsdevel. What I'm hearing is that there's no benefit to a device that can guarantee to write multiple contiguous sectors in a non-torn manner over a device that can write a single sector in a non-torn manner.

For any real benefit, filesystems need (and Linux needs to introduce plumbing for) vectored atomic writes.

That's my feeling. The non-vectored use case is pretty limited, mostly to help get a contiguous log entry on disk as a full unit. But most of the time the filesystem log commits are pretty big. Workloads with a very small number of latency sensitive writers would see improvements.

We could also the contiguous atomics for something like mysql by using a 16K sector size in the filesystem. At that point all of the db 16K units will be contig and the existing atomics proposals become interesting.

But I was really hoping for the vectors ;)

-chris

#3 lore raw
In reply to Dan Williams
Quoted 34 lines · Dan Williams
Quoted 33 lines · Chris Mason
Quoted 20 lines · Matthew Wilcox

NVMe allows the drive to tell the host what atomicity guarantees it provides for a write command. At the moment, I don't think Linux has a way for the driver to pass that information up to the filesystem.

The value that is most interesting to report is Atomic Write Unit Power Fail ("if you send a write no larger than this, the drive guarantees to write all of it or none of it"), minimum value 1 sector. [1]

There's a proposal before the NVMe workgroup to add a boundary size/offset to modify AWUPF ("except if you cross this boundary, then AWUPF is not guaranteed"). Think RAID stripe crossing.

So, three questions. Is there somewhere already to pass boundary information up to the filesystem? Can filesystems make use of a larger atomic write unit than a single sector? And, if the device is internally a RAID device, is knowing the boundary size/offset useful?

Quoted 5 lines

[1] There is also Atomic Write Unit Normal ("if you send two writes, neither of which is larger than this, subsequent reads will get either one or the other, not a mixture of both"), which I don't think we care about because the page cache prevents us from sending two writes which overlap with each other.

I think we really need the atomics to be vectored. Send N writes which as a unit are not larger than X, but which may span anywhere on device. An array with writeback cache, or a log structured squirrel in the FTL should be able to provide this pretty easily?

Quoted 7 lines
Quoted 3 lines

The immediate use case is mysql (16K writes) on a fragmented filesystem. The FS needs to be able to collect a single atomic write made up of N 4K sectors.

How big does N need to be before it starts to be generally useful? Here it seems we're talking on the order to tens of writes, but for the upper bound Dave said that N could be in the hundreds of thousands

Right, if you ask the filesystem guys, we'll want to dump the entire contents of ram down to the storage in atomic fashion. I do agree with Dave here, bigger is definitely better.

16K and up are useful, depending on which workload you're targeting. The fusion devices can do 1MB.

-chris

#4 lore raw
Quoted 41 lines · Chris Mason
Quoted 38 lines · Dan Williams
Quoted 36 lines · Chris Mason
Quoted 20 lines · Matthew Wilcox

NVMe allows the drive to tell the host what atomicity guarantees it provides for a write command. At the moment, I don't think Linux has a way for the driver to pass that information up to the filesystem.

The value that is most interesting to report is Atomic Write Unit Power Fail ("if you send a write no larger than this, the drive guarantees to write all of it or none of it"), minimum value 1 sector. [1]

There's a proposal before the NVMe workgroup to add a boundary size/offset to modify AWUPF ("except if you cross this boundary, then AWUPF is not guaranteed"). Think RAID stripe crossing.

So, three questions. Is there somewhere already to pass boundary information up to the filesystem? Can filesystems make use of a larger atomic write unit than a single sector? And, if the device is internally a RAID device, is knowing the boundary size/offset useful?

Quoted 5 lines

[1] There is also Atomic Write Unit Normal ("if you send two writes, neither of which is larger than this, subsequent reads will get either one or the other, not a mixture of both"), which I don't think we care about because the page cache prevents us from sending two writes which overlap with each other.

I think we really need the atomics to be vectored. Send N writes which as a unit are not larger than X, but which may span anywhere on device. An array with writeback cache, or a log structured squirrel in the FTL should be able to provide this pretty easily?

The immediate use case is mysql (16K writes) on a fragmented filesystem. The FS needs to be able to collect a single atomic write made up of N 4K sectors.

Quoted 7 lines
Quoted 3 lines

How big does N need to be before it starts to be generally useful? Here it seems we're talking on the order to tens of writes, but for the upper bound Dave said that N could be in the hundreds of thousands

Right, if you ask the filesystem guys, we'll want to dump the entire contents of ram down to the storage in atomic fashion. I do agree with Dave here, bigger is definitely better.

Right, bigger is better, but what about minimum requirements?

The minimum requirement I need for converting XFS is around 4MB of discontiguous single sector IOs for the worst case event. That covers the largest *single* atomic transaction log reservation we currently make on XFS at 64k block sizes.

Quoted 2 lines

16K and up are useful, depending on which workload you're targeting. The fusion devices can do 1MB.

User data workloads, yes. The moment we start thinking about atomic filesystem metadata updates, the requirements go way, way up....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]