Skip to contentSkip to conversation

nvme: reject passthrough of driver-managed Set Features

PATCH Superseded Chao Shi 4 participants 8 messages May 23 – Jul 15 NVMe

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Chao Shi

author
#0 lore raw

PATCH v5 1/1 nvme: reject passthrough of driver-managed Set Features

Since commit b58da2d270db ("nvme: update keep alive interval when kato is modified"), userspace can start keep-alive on any transport via a Set Features (KATO) passthrough command. nvme_keep_alive_work() then allocates with BLK_MQ_REQ_RESERVED, but nvme_alloc_admin_tag_set() only reserves admin tags for fabrics, so the allocation trips WARN_ON_ONCE() in blk_mq_get_tag() and fails:

nvme nvme0: keep-alive failed: -11

More generally, several Set Features change controller state that the driver manages itself and cannot react to correctly when set behind its back from userspace. Reject these in nvme_cmd_allowed():

- KATO on non-fabrics (keep-alive is only armed for fabrics; on PCIe it has no reserved tag and an active keep-alive harms idle power states)
- Host Behavior Support, Host Memory Buffer, Number of Queues, and Autonomous Power State Transition (all driver-managed)

Keep Alive on fabrics is unchanged. I/O commands are unaffected as the check is confined to the admin path (ns == NULL).

Fixes b58da2d270db ("nvme: update keep alive interval when kato is modified")

Found by FuzzNvme(Syzkaller with FEMU fuzzing framework).

---

Reproducer for the keep-alive case (run as root on a PCIe NVMe device):

    #include <fcntl.h>
    #include <stdio.h>
    #include <string.h>
    #include <sys/ioctl.h>
    #include <linux/nvme_ioctl.h>

    int main(void)
    {
            struct nvme_admin_cmd cmd = {0};
            int fd = open("/dev/nvme0", O_RDWR);
            if (fd < 0) { perror("open"); return 1; }
            cmd.opcode = 0x09;       /* SET_FEATURES */
            cmd.cdw10  = 0x0f;       /* Feature ID: KATO */
            cmd.cdw11  = 5;          /* KATO = 5 seconds */
            if (ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd) < 0) {
                    perror("ioctl");
                    return 1;
            }
            return 0;
    }

On an unpatched kernel, within ~kato/2 seconds after the program exits, dmesg shows:

nvme nvme0: keep alive interval updated from 0 ms to 5000 ms
WARNING: CPU: 0 PID: ... at block/blk-mq-tag.c:148 blk_mq_get_tag+... nvme nvme0: keep-alive failed: -11

With this patch the ioctl fails with EACCES on non-fabrics.

Changes since v4:
- Fold the check into the existing nvme_cmd_allowed() instead of a separate helper, and reject additional driver-managed Set Features (Host Behavior, Host Memory Buffer, Number of Queues, Autonomous Power State Transition) in the same switch (Keith Busch). The admin vs I/O distinction is now structural: the switch lives in the ns == NULL branch, so I/O commands (e.g. Dataset Management, which shares opcode 0x09 with Set Features) are never inspected.

Changes since v3:
- Only inspect admin commands so a DSM I/O command is not wrongly rejected (Keith Busch).

Changes since v2:
- Reject the KATO passthrough on non-fabrics instead of reserving an admin tag for all transports (Keith Busch, Christoph Hellwig).

Changes since v1:
- v2 added a spec citation and quirk discussion, superseded by the reject approach.

 drivers/nvme/host/ioctl.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
drivers/nvme/host/ioctl.c +27 -6
index a9c097dacad6..31784506e845 100644
@@ -14,8 +14,9 @@ enum {
 	NVME_IOCTL_PARTITION	= (1 << 1),
 };
 
-static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
-		unsigned int flags, bool open_for_write)
+static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			     struct nvme_command *c, unsigned int flags,
+			     bool open_for_write)
 {
 	u32 effects;
 
@@ -50,6 +51,26 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
 			case NVME_ID_CNS_CTRL:
 				return true;
 			}
+		} else if (c->common.opcode == nvme_admin_set_features) {
+			/*
+			 * Reject Set Features that change controller state the
+			 * driver manages itself; setting them behind the driver's
+			 * back from userspace leaves it unable to react correctly.
+			 * Keep Alive is only armed for fabrics - on other
+			 * transports it has no reserved tag and harms idle power
+			 * states.
+			 */
+			switch (le32_to_cpu(c->features.fid) & 0xff) {
+			case NVME_FEAT_KATO:
+				if (ctrl->ops->flags & NVME_F_FABRICS)
+					break;
+				fallthrough;
+			case NVME_FEAT_HOST_BEHAVIOR:
+			case NVME_FEAT_HOST_MEM_BUF:
+			case NVME_FEAT_NUM_QUEUES:
+			case NVME_FEAT_AUTO_PST:
+				return false;
+			}
 		}
 		goto admin;
 	}
@@ -59,7 +80,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
 	 * and marks this command as supported.  If not reject unprivileged
 	 * passthrough.
 	 */
-	effects = nvme_command_effects(ns->ctrl, ns, c->common.opcode);
+	effects = nvme_command_effects(ctrl, ns, c->common.opcode);
 	if (!(effects & NVME_CMD_EFFECTS_CSUPP))
 		goto admin;
 
@@ -308,7 +329,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.cdw14 = cpu_to_le32(cmd.cdw14);
 	c.common.cdw15 = cpu_to_le32(cmd.cdw15);
 
-	if (!nvme_cmd_allowed(ns, &c, 0, open_for_write))
+	if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write))
 		return -EACCES;
 
 	if (cmd.timeout_ms)
@@ -355,7 +376,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.cdw14 = cpu_to_le32(cmd.cdw14);
 	c.common.cdw15 = cpu_to_le32(cmd.cdw15);
 
-	if (!nvme_cmd_allowed(ns, &c, flags, open_for_write))
+	if (!nvme_cmd_allowed(ctrl, ns, &c, flags, open_for_write))
 		return -EACCES;
 
 	if (cmd.timeout_ms)
@@ -472,7 +493,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.cdw14 = cpu_to_le32(READ_ONCE(cmd->cdw14));
 	c.common.cdw15 = cpu_to_le32(READ_ONCE(cmd->cdw15));
 
-	if (!nvme_cmd_allowed(ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE))
+	if (!nvme_cmd_allowed(ctrl, ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE))
 		return -EACCES;
 
 	d.metadata = READ_ONCE(cmd->metadata);
Signature · 1 line
2.43.0
#1 lore raw
Quoted 95 lines · Chao Shi

Since commit b58da2d270db ("nvme: update keep alive interval when kato is modified"), userspace can start keep-alive on any transport via a Set Features (KATO) passthrough command. nvme_keep_alive_work() then allocates with BLK_MQ_REQ_RESERVED, but nvme_alloc_admin_tag_set() only reserves admin tags for fabrics, so the allocation trips WARN_ON_ONCE() in blk_mq_get_tag() and fails:

nvme nvme0: keep-alive failed: -11

More generally, several Set Features change controller state that the driver manages itself and cannot react to correctly when set behind its back from userspace. Reject these in nvme_cmd_allowed():

- KATO on non-fabrics (keep-alive is only armed for fabrics; on PCIe it has no reserved tag and an active keep-alive harms idle power states)
- Host Behavior Support, Host Memory Buffer, Number of Queues, and Autonomous Power State Transition (all driver-managed)

Keep Alive on fabrics is unchanged. I/O commands are unaffected as the check is confined to the admin path (ns == NULL).

Fixes b58da2d270db ("nvme: update keep alive interval when kato is modified")

Found by FuzzNvme(Syzkaller with FEMU fuzzing framework).

---

Reproducer for the keep-alive case (run as root on a PCIe NVMe device):

     #include <fcntl.h>
     #include <stdio.h>
     #include <string.h>
     #include <sys/ioctl.h>
     #include <linux/nvme_ioctl.h>

     int main(void)
     {
             struct nvme_admin_cmd cmd = {0};
             int fd = open("/dev/nvme0", O_RDWR);
             if (fd < 0) { perror("open"); return 1; }
             cmd.opcode = 0x09;       /* SET_FEATURES */
             cmd.cdw10  = 0x0f;       /* Feature ID: KATO */
             cmd.cdw11  = 5;          /* KATO = 5 seconds */
             if (ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd) < 0) {
                     perror("ioctl");
                     return 1;
             }
             return 0;
     }

On an unpatched kernel, within ~kato/2 seconds after the program exits, dmesg shows:

nvme nvme0: keep alive interval updated from 0 ms to 5000 ms
WARNING: CPU: 0 PID: ... at block/blk-mq-tag.c:148 blk_mq_get_tag+... nvme nvme0: keep-alive failed: -11

With this patch the ioctl fails with EACCES on non-fabrics.

Changes since v4:
- Fold the check into the existing nvme_cmd_allowed() instead of a separate helper, and reject additional driver-managed Set Features (Host Behavior, Host Memory Buffer, Number of Queues, Autonomous Power State Transition) in the same switch (Keith Busch). The admin vs I/O distinction is now structural: the switch lives in the ns == NULL branch, so I/O commands (e.g. Dataset Management, which shares opcode 0x09 with Set Features) are never inspected.

Changes since v3:
- Only inspect admin commands so a DSM I/O command is not wrongly rejected (Keith Busch).

Changes since v2:
- Reject the KATO passthrough on non-fabrics instead of reserving an admin tag for all transports (Keith Busch, Christoph Hellwig).

Changes since v1:
- v2 added a spec citation and quirk discussion, superseded by the reject approach.

  drivers/nvme/host/ioctl.c | 33 +++++++++++++++++++++++++++------
  1 file changed, 27 insertions(+), 6 deletions(-)
drivers/nvme/host/ioctl.c +0 -0
index a9c097dacad6..31784506e845 100644

drivers/nvme/host/ioctl.c enum { · line 19

Quoted 8 lines
drivers/nvme/host/ioctl.c +3 -2
@@ -14,8 +14,9 @@ enum {
  	NVME_IOCTL_PARTITION	= (1 << 1),
  };
 
-static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
-		unsigned int flags, bool open_for_write)
+static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			     struct nvme_command *c, unsigned int flags,
+			     bool open_for_write)

The struct nvme_ns does already have the struct nvme_ctrl *ctrl as a member variable as below so seems not necessary to add the function argument struct nvme_ctrl *ctrl. struct nvme_ns {

    struct list_head list;

    struct nvme_ctrl *ctrl;

maintainer
#2 lore raw
Quoted 7 lines · Tokunori Ikegami

drivers/nvme/host/ioctl.c enum { · line 19

Quoted 3 lines
drivers/nvme/host/ioctl.c +3 -0
+static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			     struct nvme_command *c, unsigned int flags,
+			     bool open_for_write)

The struct nvme_ns does already have the struct nvme_ctrl *ctrl as a member variable as below so seems not necessary to add the function argument struct nvme_ctrl *ctrl.

This filter is for admin commands, so the 'ns' parameter is NULL for this case.

#3 lore raw
Quoted 9 lines · Keith Busch
Quoted 6 lines · Tokunori Ikegami

drivers/nvme/host/ioctl.c enum { · line 19

Quoted 3 lines
drivers/nvme/host/ioctl.c +3 -0
+static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			     struct nvme_command *c, unsigned int flags,
+			     bool open_for_write)

The struct nvme_ns does already have the struct nvme_ctrl *ctrl as a member variable as below so seems not necessary to add the function argument struct nvme_ctrl *ctrl.

This filter is for admin commands, so the 'ns' parameter is NULL for this case.

Okay so how about that to use the function argument unsigned int flags instead with adding the NVME_IOCTL_FABRICS enum definition? enum {
    NVME_IOCTL_VEC        = (1 << 0),
    NVME_IOCTL_PARTITION    = (1 << 1),
};
Note: I mean as possible as we can it seems better to avoid the many function arguments for the maintainability. Thank you.

#4 lore raw
In reply to Chao Shi
Quoted 107 lines · Chao Shi

Since commit b58da2d270db ("nvme: update keep alive interval when kato is modified"), userspace can start keep-alive on any transport via a Set Features (KATO) passthrough command. nvme_keep_alive_work() then allocates with BLK_MQ_REQ_RESERVED, but nvme_alloc_admin_tag_set() only reserves admin tags for fabrics, so the allocation trips WARN_ON_ONCE() in blk_mq_get_tag() and fails:

nvme nvme0: keep-alive failed: -11

More generally, several Set Features change controller state that the driver manages itself and cannot react to correctly when set behind its back from userspace. Reject these in nvme_cmd_allowed():

- KATO on non-fabrics (keep-alive is only armed for fabrics; on PCIe it has no reserved tag and an active keep-alive harms idle power states)
- Host Behavior Support, Host Memory Buffer, Number of Queues, and Autonomous Power State Transition (all driver-managed)

Keep Alive on fabrics is unchanged. I/O commands are unaffected as the check is confined to the admin path (ns == NULL).

Fixes b58da2d270db ("nvme: update keep alive interval when kato is modified")

Found by FuzzNvme(Syzkaller with FEMU fuzzing framework).

---

Reproducer for the keep-alive case (run as root on a PCIe NVMe device):

    #include <fcntl.h>
    #include <stdio.h>
    #include <string.h>
    #include <sys/ioctl.h>
    #include <linux/nvme_ioctl.h>

    int main(void)
    {
            struct nvme_admin_cmd cmd = {0};
            int fd = open("/dev/nvme0", O_RDWR);
            if (fd < 0) { perror("open"); return 1; }
            cmd.opcode = 0x09;       /* SET_FEATURES */
            cmd.cdw10  = 0x0f;       /* Feature ID: KATO */
            cmd.cdw11  = 5;          /* KATO = 5 seconds */
            if (ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd) < 0) {
                    perror("ioctl");
                    return 1;
            }
            return 0;
    }

On an unpatched kernel, within ~kato/2 seconds after the program exits, dmesg shows:

nvme nvme0: keep alive interval updated from 0 ms to 5000 ms
WARNING: CPU: 0 PID: ... at block/blk-mq-tag.c:148 blk_mq_get_tag+... nvme nvme0: keep-alive failed: -11

With this patch the ioctl fails with EACCES on non-fabrics.

Changes since v4:
- Fold the check into the existing nvme_cmd_allowed() instead of a separate helper, and reject additional driver-managed Set Features (Host Behavior, Host Memory Buffer, Number of Queues, Autonomous Power State Transition) in the same switch (Keith Busch). The admin vs I/O distinction is now structural: the switch lives in the ns == NULL branch, so I/O commands (e.g. Dataset Management, which shares opcode 0x09 with Set Features) are never inspected.

Changes since v3:
- Only inspect admin commands so a DSM I/O command is not wrongly rejected (Keith Busch).

Changes since v2:
- Reject the KATO passthrough on non-fabrics instead of reserving an admin tag for all transports (Keith Busch, Christoph Hellwig).

Changes since v1:
- v2 added a spec citation and quirk discussion, superseded by the reject approach.

 drivers/nvme/host/ioctl.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
drivers/nvme/host/ioctl.c +3 -2
index a9c097dacad6..31784506e845 100644
@@ -14,8 +14,9 @@ enum {
 	NVME_IOCTL_PARTITION	= (1 << 1),
 };
 
-static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
-		unsigned int flags, bool open_for_write)
+static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			     struct nvme_command *c, unsigned int flags,
+			     bool open_for_write)
 {
 	u32 effects;
 

drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 58

Quoted 8 lines
drivers/nvme/host/ioctl.c +5 -0
@@ -50,6 +51,26 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
 			case NVME_ID_CNS_CTRL:
 				return true;
 			}
+		} else if (c->common.opcode == nvme_admin_set_features) {
+			/*
+			 * Reject Set Features that change controller state the
+			 * driver manages itself; setting them behind the driver's
+			 * back from userspace leaves it unable to react correctly.

Overly long lines. I suspect we're best off splitting out the admin and ns-command set specific parts of nvme_cmd_allowed into separate helpers. And maybe use a switch statement on the command as nested ifs become cumersome in the long run.

drivers/nvme/host/ioctl.c static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · line 496

Quoted 2 lines
drivers/nvme/host/ioctl.c +1 -1
-	if (!nvme_cmd_allowed(ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE))
+	if (!nvme_cmd_allowed(ctrl, ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE))

Another overly long line here.

Otherwise this looks good.

maintainer
#5 lore raw
In reply to Chao Shi

drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 71

Quoted 9 lines · Chao Shi
drivers/nvme/host/ioctl.c +9 -0
+			switch (le32_to_cpu(c->features.fid) & 0xff) {
+			case NVME_FEAT_KATO:
+				if (ctrl->ops->flags & NVME_F_FABRICS)
+					break;
+				fallthrough;
+			case NVME_FEAT_HOST_BEHAVIOR:
+			case NVME_FEAT_HOST_MEM_BUF:
+			case NVME_FEAT_NUM_QUEUES:
+			case NVME_FEAT_AUTO_PST:

I may have been overly restrictive with suggesting AUTO_PST for this filter. Messing with the other features will break something, but power state is just user policy. The driver may undo the user setting on a controller reset, but so what?

#6 lore raw
Quoted 15 lines · Keith Busch

drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 71

Quoted 9 lines · Chao Shi
drivers/nvme/host/ioctl.c +9 -0
+			switch (le32_to_cpu(c->features.fid) & 0xff) {
+			case NVME_FEAT_KATO:
+				if (ctrl->ops->flags & NVME_F_FABRICS)
+					break;
+				fallthrough;
+			case NVME_FEAT_HOST_BEHAVIOR:
+			case NVME_FEAT_HOST_MEM_BUF:
+			case NVME_FEAT_NUM_QUEUES:
+			case NVME_FEAT_AUTO_PST:

I may have been overly restrictive with suggesting AUTO_PST for this filter. Messing with the other features will break something, but power state is just user policy. The driver may undo the user setting on a controller reset, but so what?

There's really no point in allowing it. Just as we should have never allowed low-level config of any kind, but unfortunately nvme admin commands are a horrible grab all.

author
#7 lore raw

Sorry for the delay - I was analyzing crashes and preparing patches, and had a paper deadline. I'll respond promptly from now on.

Quoted 4 lines

I suspect we're best off splitting out the admin and ns-command set specific parts of nvme_cmd_allowed into separate helpers. And maybe use a switch statement on the command as nested ifs become cumbersome in the long run.

Agreed. v6:

- Split nvme_cmd_allowed() into nvme_admin_cmd_allowed(ctrl, c) and nvme_ns_cmd_allowed(ns, c, open_for_write), leaving nvme_cmd_allowed() a thin dispatcher, and switch on the command in the admin helper. Semantics unchanged.
- That fixes the long comment (it moves into the admin helper at a shallower indent). The nvme_uring_cmd_io() line is a caller, not the switch, so I'll hoist a local open_for_write there.
- ctrl stays: on the admin path ns is NULL (Keith's point), so it can't come from ns. Tokunori, a flags bit would encode the transport, but flags is about how the ioctl was invoked (vector / partition), so I'd rather keep ctrl - it lives in the admin helper, and the I/O helper uses ns.
- Keeping NVME_FEAT_AUTO_PST rejected - agreed there's no real reason to allow it.

I'll send v6 shortly.

Thanks,
Chao

Quoted 21 lines · Christoph Hellwig
Quoted 11 lines · Keith Busch

drivers/nvme/host/ioctl.c static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · line 71

Quoted 9 lines · Chao Shi
drivers/nvme/host/ioctl.c +9 -0
+                   switch (le32_to_cpu(c->features.fid) & 0xff) {
+                   case NVME_FEAT_KATO:
+                           if (ctrl->ops->flags & NVME_F_FABRICS)
+                                   break;
+                           fallthrough;
+                   case NVME_FEAT_HOST_BEHAVIOR:
+                   case NVME_FEAT_HOST_MEM_BUF:
+                   case NVME_FEAT_NUM_QUEUES:
+                   case NVME_FEAT_AUTO_PST:
Quoted 4 lines

I may have been overly restrictive with suggesting AUTO_PST for this filter. Messing with the other features will break something, but power state is just user policy. The driver may undo the user setting on a controller reset, but so what?

There's really no point in allowing it. Just as we should have never allowed low-level config of any kind, but unfortunately nvme admin commands are a horrible grab all.