NVMe command ordering

SPDK Jonas Pfefferle 4 participants 8 messages Mar 23, 2018 – Mar 27, 2018

Share & tools

Jonas Pfefferle

author

2018-03-23 09:04 UTC #0 lore raw

Hi @all,

Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements
"For all commands which are not part of a fused operation (refer to section 4.10), or for which the write size is greater than AWUN, each command is processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to commands submitted to other I/O Submission Queues"

So writes <= AWUN size are atomic but there is no ordering between the commands, e.g. write command A issued before write command B to the same LBA can be executed in order AB or BA, the only thing guaranteed is that they are executed atomically? And there is no way to enforce ordering except on the host side?

Regards,
Jonas

Andrey Kuzmin

2018-03-23 11:25 UTC #1 lore raw

Quoted 11 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:

Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements
"For all commands which are not part of a fused operation (refer to section 4.10), or for which the write size is greater than AWUN, each command is processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to commands submitted to other
I/O Submission Queues"

Quoted 7 lines

So writes <= AWUN size are atomic but there is no ordering between the commands, e.g. write command A issued before write command B to the same LBA
can be executed in order AB or BA, the only thing guaranteed is that they are executed atomically? And there is no way to enforce ordering except on the host side?

Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host software or the associated application is required to enforce that ordering above the level of the controller. " (1.3a, P. 179).

Regards,
Andrey

Quoted 11 lines

Regards,
Jonas
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Signature · 2 lines

Regards,
Andrey

Jonas Pfefferle

author

2018-03-23 11:59 UTC #2 lore raw

Thanks!

Quoted 55 lines · On Fri, 23 Mar 2018 11:25:26 +0000 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>…

Quoted 15 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements
"For all commands which are not part of a fused operation (refer to section
4.10), or for which the write size is greater than AWUN, each command is
processed as an independent entity without reference to other commands
submitted to the same I/O Submission Queue or to commands submitted to
other
I/O Submission Queues"
Quoted 11 lines
So writes <= AWUN size are atomic but there is no ordering between the
commands, e.g. write command A issued before write command B to the same
LBA
can be executed in order AB or BA, the only thing guaranteed is that they
are executed atomically? And there is no way to enforce ordering except on
the host side?
Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host
software or the associated application is required to enforce that ordering
above the level of the controller. " (1.3a, P. 179).
Regards,
Andrey
Quoted 9 lines
Regards,
Jonas
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
Signature · 2 lines
Regards,
Andrey

Jonas Pfefferle

author

2018-03-23 16:21 UTC #3 lore raw

One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."

Does that mean if, for example, command A and B were executed in order A,B they do not have to complete in this order on the CQ?

Thanks,
Jonas

Quoted 65 lines · On Fri, 23 Mar 2018 12:59:54 +0100 "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:

Thanks!
Quoted 46 lines · On Fri, 23 Mar 2018 11:25:26 +0000 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>…
Quoted 15 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements
"For all commands which are not part of a fused operation (refer to section
4.10), or for which the write size is greater than AWUN, each command is
processed as an independent entity without reference to other commands
submitted to the same I/O Submission Queue or to commands submitted to
other
I/O Submission Queues"
Quoted 11 lines
So writes <= AWUN size are atomic but there is no ordering between the
commands, e.g. write command A issued before write command B to the same
LBA
can be executed in order AB or BA, the only thing guaranteed is that they
are executed atomically? And there is no way to enforce ordering except on
the host side?
Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host
software or the associated application is required to enforce that ordering
above the level of the controller. " (1.3a, P. 179).
Regards,
Andrey
Quoted 4 lines
Regards,
Jonas
Quoted 9 lines
Quoted 5 lines
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
Signature · 2 lines
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Luse

2018-03-23 16:23 UTC #4 lore raw

Yup

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle
Sent: Friday, March 23, 2018 9:21 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering

One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."

Does that mean if, for example, command A and B were executed in order A,B they do not have to complete in this order on the CQ?

Thanks,
Jonas

Quoted 45 lines · On Fri, 23 Mar 2018 12:59:54 +0100 "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:

Thanks!
Quoted 41 lines · On Fri, 23 Mar 2018 11:25:26 +0000 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>…
Quoted 10 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements "For all commands which are not part of a fused operation (refer to section 4.10), or for which the write size is greater than AWUN, each command is processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to commands submitted to other I/O Submission Queues"
Quoted 6 lines
So writes <= AWUN size are atomic but there is no ordering between the commands, e.g. write command A issued before write command B to the same LBA can be executed in order AB or BA, the only thing guaranteed is that they are executed atomically? And there is no way to enforce ordering except on the host side?
Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host software or the associated application is required to enforce that ordering above the level of the controller. " (1.3a, P. 179).
Regards,
Andrey
Quoted 9 lines
Regards,
Jonas
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
--

Quoted 8 lines

Quoted 2 lines
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Marushak

2018-03-23 16:29 UTC #5 lore raw

Hey Jonas,

Looks like you got the answer here hopefully, but I wanted to provide 1 additional point of clarity. For HDD or SSD (or any other type of storage media), it's always been incumbent upon the application (most notably the file system) to provide the checks and balances to ensure ordering.

Thanks,
Nate

Quoted 90 lines

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, March 23, 2018 9:23 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
Yup
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle
Sent: Friday, March 23, 2018 9:21 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."
Does that mean if, for example, command A and B were executed in order A,B they do not have to complete in this order on the CQ?
Thanks,
Jonas
Quoted 45 lines · On Fri, 23 Mar 2018 12:59:54 +0100 "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
Thanks!
Quoted 41 lines · On Fri, 23 Mar 2018 11:25:26 +0000 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>…
Quoted 10 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements "For all commands which are not part of a fused operation (refer to section 4.10), or for which the write size is greater than AWUN, each command is processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to commands submitted to other I/O Submission Queues"
Quoted 6 lines
So writes <= AWUN size are atomic but there is no ordering between the commands, e.g. write command A issued before write command B to the same LBA can be executed in order AB or BA, the only thing guaranteed is that they are executed atomically? And there is no way to enforce ordering except on the host side?
Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host software or the associated application is required to enforce that ordering above the level of the controller. " (1.3a, P. 179).
Regards,
Andrey
Quoted 9 lines
Regards,
Jonas
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
--
Quoted 8 lines
Quoted 2 lines
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Paul Von-Stamwitz

2018-03-24 00:08 UTC #6 lore raw

This is purely FYI:

Interesting paper from FAST '18.

Barrier-Enabled IO Stack for Flash Storage
https://www.usenix.org/conference/fast18/presentation/won

-Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Marushak, Nathan
Sent: Friday, March 23, 2018 9:29 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] NVMe command ordering

Hey Jonas,

Thanks,
Nate

Quoted 84 lines

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, March 23, 2018 9:23 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
Yup
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle
Sent: Friday, March 23, 2018 9:21 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."
Does that mean if, for example, command A and B were executed in order A,B they do not have to complete in this order on the CQ?
Thanks,
Jonas
Quoted 45 lines · On Fri, 23 Mar 2018 12:59:54 +0100 "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
Thanks!
Quoted 41 lines · On Fri, 23 Mar 2018 11:25:26 +0000 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>…
Quoted 10 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements "For all commands which are not part of a fused operation (refer to section 4.10), or for which the write size is greater than AWUN, each command is processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to commands submitted to other I/O Submission Queues"
Quoted 6 lines
So writes <= AWUN size are atomic but there is no ordering between the commands, e.g. write command A issued before write command B to the same LBA can be executed in order AB or BA, the only thing guaranteed is that they are executed atomically? And there is no way to enforce ordering except on the host side?
Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host software or the associated application is required to enforce that ordering above the level of the controller. " (1.3a, P. 179).
Regards,
Andrey
Quoted 9 lines
Regards,
Jonas
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
--
Quoted 8 lines
Quoted 2 lines
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Quoted 8 lines

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Jonas Pfefferle

author

2018-03-27 07:56 UTC #7 lore raw

Thanks for the notes/clarifications. Interesting paper.

Jonas

Quoted 138 lines · On Fri, 23 Mar 2018 17:08:47 -0700 Paul Von-Stamwitz…

This is purely FYI:
Interesting paper from FAST '18.
Barrier-Enabled IO Stack for Flash Storage
https://www.usenix.org/conference/fast18/presentation/won
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Marushak, Nathan
Sent: Friday, March 23, 2018 9:29 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] NVMe command ordering
Hey Jonas,
Looks like you got the answer here hopefully, but I wanted to provide 1 additional point of clarity. For HDD or SSD (or any other type of storage media), it's always been incumbent upon the application (most notably the file system) to provide the checks and balances to ensure ordering.
Thanks,
Nate
Quoted 95 lines
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul
E
Sent: Friday, March 23, 2018 9:23 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
Yup
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle
Sent: Friday, March 23, 2018 9:21 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."
Does that mean if, for example, command A and B were executed in order
A,B they do not have to complete in this order on the CQ?
Thanks,
Jonas
Quoted 16 lines · On Fri, 23 Mar 2018 12:59:54 +0100 "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
Thanks!
Quoted 12 lines · On Fri, 23 Mar 2018 11:25:26 +0000 Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>…
Quoted 9 lines · On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements "For all commands which are not part of a fused operation (refer to section 4.10), or for which the write size is greater than AWUN, each command is
processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to
commands
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
submitted to other I/O Submission Queues"
So writes <= AWUN size are atomic but there is no ordering
between
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the commands, e.g. write command A issued before write command B to the same LBA can be executed in order AB or BA, the only
thing
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
guaranteed is that they are executed atomically? And there is no way to enforce ordering except on the host side?
Right, that's exactly what the spec says in that same paragraph
you
Quoted 2 lines
Quoted 2 lines
quote
above: "If there are ordering requirements between these
commands,
Quoted 1 line
Quoted 1 line
host software or the associated application is required to
enforce
Quoted 1 line
Quoted 1 line
that ordering above the level of the controller. " (1.3a, P.
179).
Quoted 16 lines
Quoted 16 lines
Regards,
Andrey
Quoted 9 lines
Regards,
Jonas
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
--
Quoted 8 lines
Quoted 2 lines
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
Quoted 8 lines
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk