Skip to content

nvme: In-kernel TLS support for TCP

Hannes Reinecke linux-nvme netdev 11 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v11 Latest18 patches 14 repliesIn review
  2. v1018 patches 8 repliesSuperseded
  3. v8 posting 2/217 patches 14 repliesSuperseded
Earlier versions · 8
VersionPostedPatchesRepliesStatus
v8 posting 1/217 patches7 repliesSuperseded
v717 patches6 repliesSuperseded
v616 patches19 repliesSuperseded
v514 patches30 repliesSuperseded
v417 patches29 repliesSuperseded
v318 patches30 repliesSuperseded
v218 patches50 repliesSuperseded
v1 RFC18 patches68 repliesSuperseded

Changes to v10 from the cover letter

- Include reviews from Sagi

Patches in v11 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/18nvme-keyring: register '.nvme' keyringdrivers/nvme/common/Kconfig drivers/nvme/common/Makefile drivers/nvme/common/keyring.c +2 more+82 -30 replies
2/18nvme-keyring: define a 'psk' keytypedrivers/nvme/common/keyring.c+94 -00 replies
3/18nvme: add TCP TSAS definitionsinclude/linux/nvme.h+10 -00 replies
4/18nvme-tcp: add definitions for TLS cipher suitesinclude/linux/nvme-tcp.h+6 -00 replies
5/18nvme-keyring: implement nvme_tls_psk_default()drivers/nvme/common/keyring.c include/linux/nvme-keyring.h+56 -00 replies
6/18security/keys: export key_lookup()include/linux/key.h security/keys/key.c+2 -00 replies
7/18nvme-tcp: allocate socket filedrivers/nvme/host/tcp.c+11 -20 replies
8/18nvme-tcp: enable TLS handshake upcalldrivers/nvme/host/Kconfig drivers/nvme/host/core.c drivers/nvme/host/fabrics.c +4 more+185 -70 replies
9/18nvme-tcp: control message handling for recvmsg()drivers/nvme/host/tcp.c+16 -10 replies
10/18nvme-tcp: improve icreq/icresp loggingdrivers/nvme/host/tcp.c+8 -20 replies
11/18nvme-fabrics: parse options 'keyring' and 'tls_key'drivers/nvme/host/fabrics.c drivers/nvme/host/fabrics.h drivers/nvme/host/tcp.c+70 -52 replies
12/18nvmet: make TCP sectype settable via configfsdrivers/nvme/target/configfs.c+74 -10 replies
13/18nvmet-tcp: make nvmet_tcp_alloc_queue() a void functiondrivers/nvme/target/tcp.c+10 -100 replies
14/18nvmet-tcp: allocate socket filedrivers/nvme/target/tcp.c+14 -40 replies
15/18nvmet: Set 'TREQ' to 'required' when TLS is enableddrivers/nvme/target/configfs.c drivers/nvme/target/nvmet.h+18 -20 replies
16/18nvmet-tcp: enable TLS handshake upcalldrivers/nvme/target/Kconfig drivers/nvme/target/configfs.c drivers/nvme/target/nvmet.h +1 more+187 -41 reply
17/18nvmet-tcp: control messages for recvmsg()drivers/nvme/target/tcp.c+80 -130 replies
18/18nvmet-tcp: peek icreq before starting TLSdrivers/nvme/target/configfs.c drivers/nvme/target/nvmet.h drivers/nvme/target/tcp.c+82 -93 replies

Reviews trailers per patch

17 of 18 patches have a Reviewed-by or Acked-by.

PatchSagi GrimbergDavid HowellsNitesh Shetty
1/18R-b
2/18R-b
3/18R-b
4/18R-b
5/18R-b
6/18R-b A-b
7/18R-b
8/18R-b
9/18R-b
10/18R-b
11/18R-b
12/18R-b
13/18R-b R-b
14/18R-b
15/18R-b
16/18R-b
17/18R-b
18/18

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v11 thread

Cover letter [PATCHv11 00/18] nvme: In-kernel TLS support for TCP

Hi all,

with the merge of Chuck Levers handshake upcall mechanism and my tls_read_sock() implementation finally merged with net-next it's now time to restart on the actual issue, namely implementing in-kernel TLS support for nvme-tcp.

The patchset is based on the recent net-next git tree; everything after commit ba4a734e1aa0 ("net/tls: avoid TCP window full during ->read_sock()") should work.
You might want to add the patch
'nvmet-tcp: use 'spin_lock_bh' for state_lock()' before applying this patchset; otherwise results might be ... interesting.

It also requires the 'tlshd' userspace daemon
(https://github.com/oracle/ktls-utils)
for the actual TLS handshake.
Changes for nvme-cli are already included in the upstream repository.

Theory of operation:
A dedicated '.nvme' keyring is created to hold the pre-shared keys (PSKs) for the TLS handshake. Keys will have to be provisioned before TLS handshake is attempted; that can be done with the 'nvme gen-tls-key' command for nvme-cli (patches are already merged upstream).
After connection to the remote TCP port the client side will use the 'best' PSK (as inferred from the NVMe TCP spec) or the PSK specified by the '--tls_key' option to nvme-cli and call the TLS userspace daemon to initiate a TLS handshake.
The server side will then invoke the TLS userspace daemon to run the TLS handshake.
If the TLS handshake succeeds the userspace daemon will be activating kTLS on the socket, and control is passed back to the kernel.

This implementation currently does not implement the so-called 'secure concatenation' mode from NVMe-TCP; a TPAR is still pending with fixes for it, so I'll wait until it's published before posting patches for that.

Patchset can be found at:
git.kernel.org/pub/scm/linux/kernel/git/hare/nvme.git branch tls.v16

For testing I'm using this script, running on a nvme target with NQN 'nqn.test' and using 127.0.0.1 as a port; the port has to set 'addr_tsas' to 'tls1.3':

modprobe nvmet-tcp
nvmetcli restore
modprobe nvme-tcp
./nvme gen-tls-key --subsysnqn=nqn.test -i
./nvme gen-tls-key --subsysnqn=nqn.2014-08.org.nvmexpress.discovery -i tlshd -c /etc/tlshd.conf

and then one can do a simple:

# nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.test --tls
to start the connection.

As usual, comments and reviews are welcome.

Changes to v10:
- Include reviews from Sagi

Changes to v9:
- Close race between done() and timeout()
- Add logging message for icreq/icresp failure
- Sparse fixes
- Restrict TREQ setting to 'not required' or 'required' when TLS is enabled

Changes to v8:
- Simplify reference counting as suggested by Sagi
- Implement nvmf_parse_key() to simplify options parsing
- Add patch to peek at icreq to figure out whether TLS should be started

Changes to v7:
- Include reviews from Simon
- Do not call sock_release() when sock_alloc_file() fails

Changes to v6:
- More reviews from Sagi
- Fixup non-tls connections
- Fixup nvme options handling
- Add reference counting to nvmet-tcp queues

Changes to v5:
- Include reviews from Sagi
- Split off nvmet tsas/treq handling
- Sanitize sock_file handling

Changes to v4:
- Split off network patches into a separate
patchset
- Handle TLS Alert notifications

Changes to v3:
- Really handle MSG_EOR for TLS
- Fixup MSG_SENDPAGE_NOTLAST handling
- Conditionally disable fabric option

Changes to v2:
- Included reviews from Sagi
- Removed MSG_SENDPAGE_NOTLAST
- Improved MSG_EOR handling for TLS
- Add config options NVME_TCP_TLS
and NVME_TARGET_TCP_TLS

Changes to the original RFC:
- Add a CONFIG_NVME_TLS config option
- Use a single PSK for the TLS handshake
- Make TLS connections mandatory
- Do not peek messages for the server
- Simplify data_ready callback
- Implement read_sock() for TLS

Hannes Reinecke (18):
nvme-keyring: register '.nvme' keyring
nvme-keyring: define a 'psk' keytype
nvme: add TCP TSAS definitions
nvme-tcp: add definitions for TLS cipher suites
nvme-keyring: implement nvme_tls_psk_default() security/keys: export key_lookup()
nvme-tcp: allocate socket file
nvme-tcp: enable TLS handshake upcall
nvme-tcp: control message handling for recvmsg()
nvme-tcp: improve icreq/icresp logging
nvme-fabrics: parse options 'keyring' and 'tls_key'
nvmet: make TCP sectype settable via configfs
nvmet-tcp: make nvmet_tcp_alloc_queue() a void function
nvmet-tcp: allocate socket file
nvmet: Set 'TREQ' to 'required' when TLS is enabled
nvmet-tcp: enable TLS handshake upcall
nvmet-tcp: control messages for recvmsg()
nvmet-tcp: peek icreq before starting TLS

 drivers/nvme/common/Kconfig    |   4 +
 drivers/nvme/common/Makefile   |   3 +-
 drivers/nvme/common/keyring.c  | 182 ++++++++++++++++++
 drivers/nvme/host/Kconfig      |  15 ++
 drivers/nvme/host/core.c       |  12 +-
 drivers/nvme/host/fabrics.c    |  67 ++++++-
 drivers/nvme/host/fabrics.h    |   9 +
 drivers/nvme/host/nvme.h       |   1 +
 drivers/nvme/host/sysfs.c      |  21 +++
 drivers/nvme/host/tcp.c        | 184 ++++++++++++++++--
 drivers/nvme/target/Kconfig    |  15 ++
 drivers/nvme/target/configfs.c | 128 ++++++++++++-
 drivers/nvme/target/nvmet.h    |  11 ++
 drivers/nvme/target/tcp.c      | 334 ++++++++++++++++++++++++++++++---
 include/linux/key.h            |   1 +
 include/linux/nvme-keyring.h   |  36 ++++
 include/linux/nvme-tcp.h       |   6 +
 include/linux/nvme.h           |  10 +
 security/keys/key.c            |   1 +
 19 files changed, 991 insertions(+), 49 deletions(-)
 create mode 100644 drivers/nvme/common/keyring.c
 create mode 100644 include/linux/nvme-keyring.h
Signature · 1 line
2.35.3