Skip to content

nvmet: FDP bridge for NVMe-oF targets

Guixin Liu linux-nvme 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest12 patches 4 repliesIn review
  2. v211 patches 4 repliesSuperseded
  3. v1 posting 2/211 patches 4 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v1 posting 1/211 patches0 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12nvme: add the FDP wire structures for the I/O management send commandinclude/linux/nvme.h+24 -00 replies
2/12block: add bdev_write_stream_granularity()include/linux/blkdev.h+8 -00 replies
3/12nvme: export nvme_bdev_opsdrivers/nvme/host/core.c+1 -00 replies
4/12nvmet: add per-namespace FDP bridge state and configurationdrivers/nvme/target/Kconfig drivers/nvme/target/Makefile drivers/nvme/target/configfs.c +3 more+165 -10 replies
5/12nvmet: document the fdp_enable configfs attributeDocumentation/ABI/stable/configfs-nvmet+13 -00 replies
6/12nvmet: report the Endurance Group Identifier in the CS-INDEP identify datadrivers/nvme/target/admin-cmd.c+1 -00 replies
7/12nvmet: advertise the FDP capabilitydrivers/nvme/target/admin-cmd.c+2 -00 replies
8/12nvmet: implement the FDP featuredrivers/nvme/target/admin-cmd.c drivers/nvme/target/fdp.c drivers/nvme/target/nvmet.h+130 -00 replies
9/12nvmet: forward the FDP I/O management commands to the backing devicedrivers/nvme/target/admin-cmd.c drivers/nvme/target/core.c drivers/nvme/target/fdp.c +1 more+418 -60 replies
10/12nvmet: serve the FDP log pagesdrivers/nvme/target/admin-cmd.c drivers/nvme/target/fdp.c drivers/nvme/target/nvmet.h+165 -00 replies
11/12nvmet: translate placement identifiers of writes to write streamsdrivers/nvme/target/io-cmd-bdev.c drivers/nvme/target/nvmet.h+31 -00 replies
12/12nvmet: add an FDP write translation tracepointdrivers/nvme/target/io-cmd-bdev.c drivers/nvme/target/trace.h+31 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH v3 0/12] nvmet: FDP bridge for NVMe-oF targets

Flexible Data Placement (FDP) is defined in the NVMe base specification and therefore applies to fabrics as well. An Ethernet Bunch Of Flash (EBOF) style storage system that exposes its local FDP-capable NVMe drives to remote hosts over NVMe-oF currently has no way to pass that capability through: nvmet neither advertises FDP nor translates or forwards any FDP command, so a host connected through the target loses the write amplification benefits that the local drives provide.

This series adds an FDP bridge to the NVMe target. A namespace backed by an NVMe namespace opts in through the fdp_enable configfs attribute; the bridge then advertises the FDPS capability, reports the backing FDP configuration through the feature and log pages, translates the placement identifiers of write commands to the backing write streams, and forwards the FDP management commands and log pages to the backing controller. The backing geometry is verified at namespace enable time, so the bridge only comes up when the backing placement identifiers are identity mapped, which the translation relies on.

The FDP commands are served by the bridge directly instead of relying on the existing passthru support, for several reasons:

- Passthru subsystems do not support reservations; the reservation commands are rejected and the controller is not advertised as reservation capable, because the underlying device cannot differentiate the hosts that connect over fabrics. A regular namespace goes through the target's own reservation implementation, which is not available to a passthru subsystem.

- A passthru subsystem is bound to a single backend controller: the whole drive, with all of its namespaces, is passed through as one unit, and no regular namespaces can be added to the subsystem. An EBOF system needs to compose a storage pool out of individual namespaces from multiple backend drives, which only the regular namespace path provides.

- The regular namespace path already handles the target side of reservations, ANA reporting and per-namespace configuration; the bridge extends that path rather than replacing it.

- With passthru, the namespace identity of the backend drive is exposed to the host as is, and the host would talk to the backend controller's FDP state through the passthrough of every command. The bridge instead owns the FDP presentation of the exported namespace, while the state that the backing controller owns is forwarded to it: the reclaim unit handle status hosts read and the reclaim unit handle updates they issue are carried out by the controller that actually manages the media.

The Linux host side needs no changes: the existing FDP discovery path (FDPS bit, endurance group identifier, FDP feature, FDP configurations log page and reclaim unit handle status) works against the bridge unmodified.

The series is structured as follows:

  1-2   wire structures and block layer helper
  3     nvme_bdev_ops export, used to recognize an NVMe backing device
  4     per-namespace bridge state and configuration
  5     fdp_enable ABI documentation
  6     endurance group identifier in the CS-INDEP identify data
  7     FDPS advertisement
  8     FDP feature (FID 1Dh)
  9     I/O management command forwarding
  10    FDP log pages (LID 20h-23h)
  11    placement identifier translation on the write path
  12    tracepoint for the translation

This was tested with a QEMU FDP backing device through an nvmet-tcp loopback setup, with persistent reservations enabled on the exported
namespace: the host discovers and registers the backing write streams over the bridge, writes with placement identifiers land in the reclaim units of the backend device they select (verified through the reclaim unit handle status of the backend, for single bio and chained bio writes alike), and the forwarded log pages match the backend byte for byte. Short and offset reads of the configurations log match the full page, and Set Features on a namespace without FDP accepts the unchanged disabled configuration. I/O management receive requests whose NUMD arithmetic would wrap, zero-length forwards, and requests above the 1 MiB staging bound are rejected with Invalid Field. A second namespace without FDP in the same subsystem comes up normally and reports FDPE clear. With a second controller registered, the FDP commands are served while a write exclusive reservation is held, writes from the non-holder conflict, and preempt-and-abort completes with I/O management commands in flight.

Changes since v2:

https://lore.kernel.org/all/[email protected]/

- Take the namespace reference of the FDP commands under NVMET_NS_IO_LIVE with percpu_ref_tryget_live_rcu(), closing a use-after-free against a concurrently disabled namespace.
- Report FDPE clear from Get Features for namespaces without FDP, so that a subsystem mixing bridged and regular namespaces does not fail the host's per-namespace discovery; Set Features accepts that same disabled configuration as a no-op.
- Evaluate the I/O management receive transfer length in 64 bits instead of letting the 32-bit NUMD arithmetic wrap, bound all staged FDP transfers to 1 MiB, and reject zero-length forwards.
- Re-encode NUMD of the forwarded log pages from the staged transfer length instead of carrying over the host supplied value, and clear the reserved metadata field of the forwarded commands.
- Fall through to the persistent reservation access checks for the I/O management commands instead of returning before them.
- Require the backing reclaim unit handle count to match the write streams exactly instead of accepting a larger count.
- Read the backing FDP configurations log at enable time so the advertised handle types and the 64-bit reclaim unit nominal size are the backing device's own values, selected by the active FDPCIDX rather than by descriptor order, and validate every descriptor before dereferencing it.
- Serve short reads and nonzero Log Page Offsets of the configurations log as windows of the complete page.
- Run the FDP command handlers from a workqueue so that lpfc cannot sleep in hardirq context.
- Track the FDP capability as a per-subsystem count, incremented at namespace enable and decremented at namespace disable.
- Advertise the I/O management commands in the command-effects log only when the subsystem actually bridges FDP, and set error_loc for invalid FDP feature requests.
- Document fdp_enable in the ABI file.

Changes since v1:

https://lore.kernel.org/all/[email protected]/

- Verify the transfer length encoded in the I/O management commands against the length the host supplied, and do not carry the host data pointer over to the backing command.
- Assign the translated write stream to every bio of a write, not just the first one.
- Run the namespace lookup of the FDP commands under RCU.
- Report the FDP feature in the Features log page under the same per-subsystem condition as the FDPS bit.
- Only emit the write translation tracepoint for writes that carry the Data Placement Directive.
- Dispatch the I/O management commands after the persistent reservation checks.
- Report backing command failures during the namespace enable as errno, and clean up the backend state when the enable fails.
- Drop the Assisted-by: LLM tag from all commits.

Guixin Liu (12):
nvme: add the FDP wire structures for the I/O management send command
block: add bdev_write_stream_granularity()
nvme: export nvme_bdev_ops
nvmet: add per-namespace FDP bridge state and configuration
nvmet: document the fdp_enable configfs attribute
nvmet: report the Endurance Group Identifier in the CS-INDEP identify
data
nvmet: advertise the FDP capability
nvmet: implement the FDP feature
nvmet: forward the FDP I/O management commands to the backing device
nvmet: serve the FDP log pages
nvmet: translate placement identifiers of writes to write streams
nvmet: add an FDP write translation tracepoint

 Documentation/ABI/stable/configfs-nvmet |  13 +
 drivers/nvme/host/core.c                |   1 +
 drivers/nvme/target/Kconfig             |  14 +
 drivers/nvme/target/Makefile            |   1 +
 drivers/nvme/target/admin-cmd.c         |  45 +-
 drivers/nvme/target/configfs.c          |  32 ++
 drivers/nvme/target/core.c              |  43 +-
 drivers/nvme/target/fdp.c               | 682 ++++++++++++++++++++++++
 drivers/nvme/target/io-cmd-bdev.c       |  15 +
 drivers/nvme/target/nvmet.h             |  91 ++++
 drivers/nvme/target/trace.h             |  23 +
 include/linux/blkdev.h                  |   8 +
 include/linux/nvme.h                    |  24 +
 13 files changed, 987 insertions(+), 5 deletions(-)
 create mode 100644 drivers/nvme/target/fdp.c
Signature · 1 line
2.43.7