nvme: reject passthrough of driver-managed Set Features
PATCH v5 1/1 nvme: reject passthrough of driver-managed Set Features
Since commit b58da2d270db ("nvme: update keep alive interval when kato is modified"), userspace can start keep-alive on any transport via a Set Features (KATO) passthrough command. nvme_keep_alive_work() then allocates with BLK_MQ_REQ_RESERVED, but nvme_alloc_admin_tag_set() only reserves admin tags for fabrics, so the allocation trips WARN_ON_ONCE() in blk_mq_get_tag() and fails:
nvme nvme0: keep-alive failed: -11
More generally, several Set Features change controller state that the driver manages itself and cannot react to correctly when set behind its back from userspace. Reject these in nvme_cmd_allowed():
- KATO on non-fabrics (keep-alive is only armed for fabrics; on PCIe it has no reserved tag and an active keep-alive harms idle power states)
- Host Behavior Support, Host Memory Buffer, Number of Queues, and Autonomous Power State Transition (all driver-managed)
Keep Alive on fabrics is unchanged. I/O commands are unaffected as the check is confined to the admin path (ns == NULL).
Found by FuzzNvme(Syzkaller with FEMU fuzzing framework).
---
Reproducer for the keep-alive case (run as root on a PCIe NVMe device):
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/nvme_ioctl.h>
int main(void)
{
struct nvme_admin_cmd cmd = {0};
int fd = open("/dev/nvme0", O_RDWR);
if (fd < 0) { perror("open"); return 1; }
cmd.opcode = 0x09; /* SET_FEATURES */
cmd.cdw10 = 0x0f; /* Feature ID: KATO */
cmd.cdw11 = 5; /* KATO = 5 seconds */
if (ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd) < 0) {
perror("ioctl");
return 1;
}
return 0;
}On an unpatched kernel, within ~kato/2 seconds after the program exits, dmesg shows:
nvme nvme0: keep alive interval updated from 0 ms to 5000 ms
WARNING: CPU: 0 PID: ... at block/blk-mq-tag.c:148 blk_mq_get_tag+... nvme nvme0: keep-alive failed: -11
With this patch the ioctl fails with EACCES on non-fabrics.
Changes since v4:
- Fold the check into the existing nvme_cmd_allowed() instead of a separate helper, and reject additional driver-managed Set Features (Host Behavior, Host Memory Buffer, Number of Queues, Autonomous Power State Transition) in the same switch (Keith Busch). The admin vs I/O distinction is now structural: the switch lives in the ns == NULL branch, so I/O commands (e.g. Dataset Management, which shares opcode 0x09 with Set Features) are never inspected.
Changes since v3:
- Only inspect admin commands so a DSM I/O command is not wrongly rejected (Keith Busch).
Changes since v2:
- Reject the KATO passthrough on non-fabrics instead of reserving an admin tag for all transports (Keith Busch, Christoph Hellwig).
Changes since v1:
- v2 added a spec citation and quirk discussion, superseded by the reject approach.
drivers/nvme/host/ioctl.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)
drivers/nvme/host/ioctl.c +27 -6
@@ -14,8 +14,9 @@ enum { NVME_IOCTL_PARTITION = (1 << 1), }; -static bool nvme_cmd_allowed (struct nvme_ns *ns,struct nvme_command *c, -unsigned int flags,bool open_for_write) +static bool nvme_cmd_allowed (struct nvme_ctrl *ctrl,struct nvme_ns *ns, +struct nvme_command *c,unsigned int flags, +bool open_for_write) {u32 effects;
@@ -50,6 +51,26 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,case NVME_ID_CNS_CTRL :return true; } + }else if (c->common.opcode == nvme_admin_set_features) { +/* +* Reject Set Features that change controller state the +* driver manages itself; setting them behind the driver's +* back from userspace leaves it unable to react correctly. +* Keep Alive is only armed for fabrics - on other +* transports it has no reserved tag and harms idle power +* states. +*/ +switch (le32_to_cpu (c->features.fid) &0xff ) { +case NVME_FEAT_KATO : +if (ctrl->ops->flags &NVME_F_FABRICS ) +break ; + fallthrough; +case NVME_FEAT_HOST_BEHAVIOR : +case NVME_FEAT_HOST_MEM_BUF : +case NVME_FEAT_NUM_QUEUES : +case NVME_FEAT_AUTO_PST : +return false; + } } goto admin; }
@@ -59,7 +80,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,* and marks this command as supported. If not reject unprivileged * passthrough. */ - effects =nvme_command_effects (ns->ctrl, ns, c->common.opcode); + effects =nvme_command_effects (ctrl, ns, c->common.opcode);if (!(effects &NVME_CMD_EFFECTS_CSUPP )) goto admin;
@@ -308,7 +329,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, c.common.cdw14 =cpu_to_le32 (cmd.cdw14); c.common.cdw15 =cpu_to_le32 (cmd.cdw15); -if (!nvme_cmd_allowed (ns, &c,0 , open_for_write)) +if (!nvme_cmd_allowed (ctrl, ns, &c,0 , open_for_write))return -EACCES ;if (cmd.timeout_ms)
@@ -355,7 +376,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, c.common.cdw14 =cpu_to_le32 (cmd.cdw14); c.common.cdw15 =cpu_to_le32 (cmd.cdw15); -if (!nvme_cmd_allowed (ns, &c, flags, open_for_write)) +if (!nvme_cmd_allowed (ctrl, ns, &c, flags, open_for_write))return -EACCES ;if (cmd.timeout_ms)
@@ -472,7 +493,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, c.common.cdw14 =cpu_to_le32 (READ_ONCE (cmd->cdw14)); c.common.cdw15 =cpu_to_le32 (READ_ONCE (cmd->cdw15)); -if (!nvme_cmd_allowed (ns, &c,0 , ioucmd->file->f_mode &FMODE_WRITE )) +if (!nvme_cmd_allowed (ctrl, ns, &c,0 , ioucmd->file->f_mode &FMODE_WRITE ))return -EACCES ; d.metadata =READ_ONCE (cmd->metadata);
Signature · 1 line
2.43.0
Quoted 95 lines · Chao Shi
Since commit
b58da2d270db("nvme: update keep alive interval when kato is modified"), userspace can start keep-alive on any transport via a Set Features (KATO) passthrough command.nvme_keep_alive_work()then allocates withBLK_MQ_REQ_RESERVED, butnvme_alloc_admin_tag_set()only reserves admin tags for fabrics, so the allocation tripsWARN_ON_ONCE()inblk_mq_get_tag()and fails:nvme nvme0: keep-alive failed: -11
More generally, several Set Features change controller state that the driver manages itself and cannot react to correctly when set behind its back from userspace. Reject these in
nvme_cmd_allowed():- KATO on non-fabrics (keep-alive is only armed for fabrics; on PCIe it has no reserved tag and an active keep-alive harms idle power states)
- Host Behavior Support, Host Memory Buffer, Number of Queues, and Autonomous Power State Transition (all driver-managed)Keep Alive on fabrics is unchanged. I/O commands are unaffected as the check is confined to the admin path (ns == NULL).
Fixes b58da2d270db ("nvme: update keep alive interval when kato is modified")Found by FuzzNvme(Syzkaller with FEMU fuzzing framework).
Acked-by Sungwoo Kim <[email protected]>Acked-by Dave Tian <[email protected]>Acked-by Weidong Zhu <[email protected]>Signed-off-by Chao Shi <[email protected]>---
Reproducer for the keep-alive case (run as root on a PCIe NVMe device):
#include <fcntl.h> #include <stdio.h> #include <string.h> #include <sys/ioctl.h> #include <linux/nvme_ioctl.h> int main(void) { struct nvme_admin_cmd cmd = {0}; int fd = open("/dev/nvme0", O_RDWR); if (fd < 0) { perror("open"); return 1; } cmd.opcode = 0x09; /* SET_FEATURES */ cmd.cdw10 = 0x0f; /* Feature ID: KATO */ cmd.cdw11 = 5; /* KATO = 5 seconds */ if (ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd) < 0) { perror("ioctl"); return 1; } return 0; }On an unpatched kernel, within ~kato/2 seconds after the program exits, dmesg shows:
nvme nvme0: keep alive interval updated from 0 ms to 5000 ms
WARNING: CPU: 0 PID: ... atblock/blk-mq-tag.c:148blk_mq_get_tag+... nvme nvme0: keep-alive failed: -11With this patch the ioctl fails with EACCES on non-fabrics.
Changes since v4:
- Fold the check into the existingnvme_cmd_allowed()instead of a separate helper, and reject additional driver-managed Set Features (Host Behavior, Host Memory Buffer, Number of Queues, Autonomous Power State Transition) in the same switch (Keith Busch). The admin vs I/O distinction is now structural: the switch lives in the ns == NULL branch, so I/O commands (e.g. Dataset Management, which shares opcode0x09with Set Features) are never inspected.Changes since v3:
- Only inspect admin commands so a DSM I/O command is not wrongly rejected (Keith Busch).Changes since v2:
- Reject the KATO passthrough on non-fabrics instead of reserving an admin tag for all transports (Keith Busch, Christoph Hellwig).Changes since v1:
- v2 added a spec citation and quirk discussion, superseded by the reject approach.drivers/nvme/host/ioctl.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)drivers/nvme/host/ioctl.c +0
-0
drivers/nvme/host/ioctl.c enum { · line 19
Quoted 8 lines
drivers/nvme/host/ioctl.c +3
-2@@ -14,8 +14,9 @@ enum { NVME_IOCTL_PARTITION = (1 << 1), }; -static bool nvme_cmd_allowed (struct nvme_ns *ns,struct nvme_command *c, -unsigned int flags,bool open_for_write) +static bool nvme_cmd_allowed (struct nvme_ctrl *ctrl,struct nvme_ns *ns, +struct nvme_command *c,unsigned int flags, +bool open_for_write)
The struct nvme_ns does already have the struct nvme_ctrl *ctrl as a member variable as below so seems not necessary to add the function argument struct nvme_ctrl *ctrl. struct nvme_ns {
struct list_head list;
struct nvme_ctrl *ctrl;
Keith Busch
maintainerQuoted 7 lines · Tokunori Ikegami
drivers/nvme/host/ioctl.c enum { · line 19
Quoted 3 lines
drivers/nvme/host/ioctl.c +3
-0+static bool nvme_cmd_allowed (struct nvme_ctrl *ctrl,struct nvme_ns *ns, +struct nvme_command *c,unsigned int flags, +bool open_for_write)The struct
nvme_nsdoes already have the structnvme_ctrl*ctrl as a member variable as below so seems not necessary to add the function argument structnvme_ctrl*ctrl.
This filter is for admin commands, so the 'ns' parameter is NULL for this case.
Quoted 9 lines · Keith Busch
Quoted 6 lines · Tokunori Ikegami
drivers/nvme/host/ioctl.c enum { · line 19
Quoted 3 lines
drivers/nvme/host/ioctl.c +3
-0+static bool nvme_cmd_allowed (struct nvme_ctrl *ctrl,struct nvme_ns *ns, +struct nvme_command *c,unsigned int flags, +bool open_for_write)The struct
nvme_nsdoes already have the structnvme_ctrl*ctrl as a member variable as below so seems not necessary to add the function argument structnvme_ctrl*ctrl.This filter is for admin commands, so the 'ns' parameter is NULL for this case.
Okay so how about that to use the function argument unsigned int flags instead with adding the NVME_IOCTL_FABRICS enum definition? enum {
NVME_IOCTL_VEC = (1 << 0),
NVME_IOCTL_PARTITION = (1 << 1),
};
Note: I mean as possible as we can it seems better to avoid the many function arguments for the maintainability. Thank you.
Christoph Hellwig
maintainerQuoted 107 lines · Chao Shi
Since commit
b58da2d270db("nvme: update keep alive interval when kato is modified"), userspace can start keep-alive on any transport via a Set Features (KATO) passthrough command.nvme_keep_alive_work()then allocates withBLK_MQ_REQ_RESERVED, butnvme_alloc_admin_tag_set()only reserves admin tags for fabrics, so the allocation tripsWARN_ON_ONCE()inblk_mq_get_tag()and fails:nvme nvme0: keep-alive failed: -11
More generally, several Set Features change controller state that the driver manages itself and cannot react to correctly when set behind its back from userspace. Reject these in
nvme_cmd_allowed():- KATO on non-fabrics (keep-alive is only armed for fabrics; on PCIe it has no reserved tag and an active keep-alive harms idle power states)
- Host Behavior Support, Host Memory Buffer, Number of Queues, and Autonomous Power State Transition (all driver-managed)Keep Alive on fabrics is unchanged. I/O commands are unaffected as the check is confined to the admin path (ns == NULL).
Fixes b58da2d270db ("nvme: update keep alive interval when kato is modified")Found by FuzzNvme(Syzkaller with FEMU fuzzing framework).
Acked-by Sungwoo Kim <[email protected]>Acked-by Dave Tian <[email protected]>Acked-by Weidong Zhu <[email protected]>Signed-off-by Chao Shi <[email protected]>---
Reproducer for the keep-alive case (run as root on a PCIe NVMe device):
#include <fcntl.h> #include <stdio.h> #include <string.h> #include <sys/ioctl.h> #include <linux/nvme_ioctl.h> int main(void) { struct nvme_admin_cmd cmd = {0}; int fd = open("/dev/nvme0", O_RDWR); if (fd < 0) { perror("open"); return 1; } cmd.opcode = 0x09; /* SET_FEATURES */ cmd.cdw10 = 0x0f; /* Feature ID: KATO */ cmd.cdw11 = 5; /* KATO = 5 seconds */ if (ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd) < 0) { perror("ioctl"); return 1; } return 0; }On an unpatched kernel, within ~kato/2 seconds after the program exits, dmesg shows:
nvme nvme0: keep alive interval updated from 0 ms to 5000 ms
WARNING: CPU: 0 PID: ... atblock/blk-mq-tag.c:148blk_mq_get_tag+... nvme nvme0: keep-alive failed: -11With this patch the ioctl fails with EACCES on non-fabrics.
Changes since v4:
- Fold the check into the existingnvme_cmd_allowed()instead of a separate helper, and reject additional driver-managed Set Features (Host Behavior, Host Memory Buffer, Number of Queues, Autonomous Power State Transition) in the same switch (Keith Busch). The admin vs I/O distinction is now structural: the switch lives in the ns == NULL branch, so I/O commands (e.g. Dataset Management, which shares opcode0x09with Set Features) are never inspected.Changes since v3:
- Only inspect admin commands so a DSM I/O command is not wrongly rejected (Keith Busch).Changes since v2:
- Reject the KATO passthrough on non-fabrics instead of reserving an admin tag for all transports (Keith Busch, Christoph Hellwig).Changes since v1:
- v2 added a spec citation and quirk discussion, superseded by the reject approach.drivers/nvme/host/ioctl.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)drivers/nvme/host/ioctl.c +3
-2@@ -14,8 +14,9 @@ enum { NVME_IOCTL_PARTITION = (1 << 1), }; -static bool nvme_cmd_allowed (struct nvme_ns *ns,struct nvme_command *c, -unsigned int flags,bool open_for_write) +static bool nvme_cmd_allowed (struct nvme_ctrl *ctrl,struct nvme_ns *ns, +struct nvme_command *c,unsigned int flags, +bool open_for_write) {u32 effects;
drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 58
Quoted 8 lines
drivers/nvme/host/ioctl.c +5
-0@@ -50,6 +51,26 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,case NVME_ID_CNS_CTRL :return true; } + }else if (c->common.opcode == nvme_admin_set_features) { +/* +* Reject Set Features that change controller state the +* driver manages itself; setting them behind the driver's +* back from userspace leaves it unable to react correctly.
Overly long lines. I suspect we're best off splitting out the admin and ns-command set specific parts of nvme_cmd_allowed into separate helpers. And maybe use a switch statement on the command as nested ifs become cumersome in the long run.
drivers/nvme/host/ioctl.c static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · line 496
Quoted 2 lines
drivers/nvme/host/ioctl.c +1
-1-if (!nvme_cmd_allowed (ns, &c,0 , ioucmd->file->f_mode &FMODE_WRITE )) +if (!nvme_cmd_allowed (ctrl, ns, &c,0 , ioucmd->file->f_mode &FMODE_WRITE ))
Another overly long line here.
Otherwise this looks good.
Keith Busch
maintainerdrivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 71
Quoted 9 lines · Chao Shi
drivers/nvme/host/ioctl.c +9
-0+switch (le32_to_cpu (c->features.fid) &0xff ) { +case NVME_FEAT_KATO : +if (ctrl->ops->flags &NVME_F_FABRICS ) +break ; + fallthrough; +case NVME_FEAT_HOST_BEHAVIOR : +case NVME_FEAT_HOST_MEM_BUF : +case NVME_FEAT_NUM_QUEUES : +case NVME_FEAT_AUTO_PST :
I may have been overly restrictive with suggesting AUTO_PST for this filter. Messing with the other features will break something, but power state is just user policy. The driver may undo the user setting on a controller reset, but so what?
Christoph Hellwig
maintainerQuoted 15 lines · Keith Busch
drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 71
Quoted 9 lines · Chao Shi
drivers/nvme/host/ioctl.c +9
-0+switch (le32_to_cpu (c->features.fid) &0xff ) { +case NVME_FEAT_KATO : +if (ctrl->ops->flags &NVME_F_FABRICS ) +break ; + fallthrough; +case NVME_FEAT_HOST_BEHAVIOR : +case NVME_FEAT_HOST_MEM_BUF : +case NVME_FEAT_NUM_QUEUES : +case NVME_FEAT_AUTO_PST :I may have been overly restrictive with suggesting
AUTO_PSTfor this filter. Messing with the other features will break something, but power state is just user policy. The driver may undo the user setting on a controller reset, but so what?
There's really no point in allowing it. Just as we should have never allowed low-level config of any kind, but unfortunately nvme admin commands are a horrible grab all.
Sorry for the delay - I was analyzing crashes and preparing patches, and had a paper deadline. I'll respond promptly from now on.
Quoted 4 lines
I suspect we're best off splitting out the admin and ns-command set specific parts of
nvme_cmd_allowedinto separate helpers. And maybe use a switch statement on the command as nested ifs become cumbersome in the long run.
Agreed. v6:
- Split nvme_cmd_allowed() into nvme_admin_cmd_allowed(ctrl, c) and nvme_ns_cmd_allowed(ns, c, open_for_write), leaving nvme_cmd_allowed() a thin dispatcher, and switch on the command in the admin helper. Semantics unchanged.
- That fixes the long comment (it moves into the admin helper at a shallower indent). The nvme_uring_cmd_io() line is a caller, not the switch, so I'll hoist a local open_for_write there.
- ctrl stays: on the admin path ns is NULL (Keith's point), so it can't come from ns. Tokunori, a flags bit would encode the transport, but flags is about how the ioctl was invoked (vector / partition), so I'd rather keep ctrl - it lives in the admin helper, and the I/O helper uses ns.
- Keeping NVME_FEAT_AUTO_PST rejected - agreed there's no real reason to allow it.
I'll send v6 shortly.
Thanks,
Chao
Quoted 21 lines · Christoph Hellwig
Quoted 11 lines · Keith Busch
drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 71
Quoted 9 lines · Chao Shi
drivers/nvme/host/ioctl.c +9
-0+switch (le32_to_cpu (c->features.fid) &0xff ) { +case NVME_FEAT_KATO : +if (ctrl->ops->flags &NVME_F_FABRICS ) +break ; + fallthrough; +case NVME_FEAT_HOST_BEHAVIOR : +case NVME_FEAT_HOST_MEM_BUF : +case NVME_FEAT_NUM_QUEUES : +case NVME_FEAT_AUTO_PST :Quoted 4 lines
I may have been overly restrictive with suggesting
AUTO_PSTfor this filter. Messing with the other features will break something, but power state is just user policy. The driver may undo the user setting on a controller reset, but so what?There's really no point in allowing it. Just as we should have never allowed low-level config of any kind, but unfortunately nvme admin commands are a horrible grab all.