Skip to contentSkip to conversation

qla2xxx: Fix unbound NVME response length

PATCH Applied Himanshu Madhani 5 participants 8 messages Jan 21, 2020 – Jan 23, 2020

Subsystems (2)

v4 of 5 Applied Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 qla2xxx: Fix unbound NVME response length

From: Arun Easi <[email protected]>

On certain cases when response length is less than 32, NVME response data is supplied inline in IOCB. This is indicated by some combination of state flags. There was an instance when a high, and incorrect, response length was indicated causing driver to overrun buffers. Fix this by checking and limiting the response payload length.

Fixes 7401bc18d1ee3 ("scsi: qla2xxx: Add FC-NVMe command handling")

---
Hi Martin,

We discovered issue with our newer Gen7 adapter when response length happens to be larger than 32 bytes, could result into crash.

Please apply this to 5.5/scsi-fixes branch at your earliest convenience.

Changes from v3 -> v4

o use "sizeof(struct nvme_fc_ersp_iu)" in missed place.

Changes from v2 -> v3

o Use "sizeof(struct nvme_fc_ersp_iu)" to indicate response payload size.

Changes from v1 -> v2

o Fixed the tag for stable.
o Removed logit which got spilled from other patch to prevent compile failure.

Thanks,
Himanshu

---
 drivers/scsi/qla2xxx/qla_isr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/scsi/qla2xxx/qla_isr.c +10 -0
index e7bad0bfffda..4caec94d8e99 100644
@@ -1939,6 +1939,16 @@ static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req,
 		inbuf = (uint32_t *)&sts->nvme_ersp_data;
 		outbuf = (uint32_t *)fd->rspaddr;
 		iocb->u.nvme.rsp_pyld_len = le16_to_cpu(sts->nvme_rsp_pyld_len);
+		if (unlikely(iocb->u.nvme.rsp_pyld_len >
+		    sizeof(struct nvme_fc_ersp_iu))) {
+			WARN_ONCE(1, "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);
+			ql_log(ql_log_warn, fcport->vha, 0x5100,
+			    "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);
+			iocb->u.nvme.rsp_pyld_len =
+			    sizeof(struct nvme_fc_ersp_iu);
+		}
 		iter = iocb->u.nvme.rsp_pyld_len >> 2;
 		for (; iter; iter--)
 			*outbuf++ = swab32(*inbuf++);
Signature · 1 line
2.12.0
Reviewed-by on v4 1/1
#1 lore raw
Quoted 50 lines · Himanshu Madhani

From: Arun Easi <[email protected]>

On certain cases when response length is less than 32, NVME response data is supplied inline in IOCB. This is indicated by some combination of state flags. There was an instance when a high, and incorrect, response length was indicated causing driver to overrun buffers. Fix this by checking and limiting the response payload length.

Fixes 7401bc18d1ee3 ("scsi: qla2xxx: Add FC-NVMe command handling")

---
Hi Martin,

We discovered issue with our newer Gen7 adapter when response length happens to be larger than 32 bytes, could result into crash.

Please apply this to 5.5/scsi-fixes branch at your earliest convenience.

Changes from v3 -> v4

o use "sizeof(struct nvme_fc_ersp_iu)" in missed place.

Changes from v2 -> v3

o Use "sizeof(struct nvme_fc_ersp_iu)" to indicate response payload size.

Changes from v1 -> v2

o Fixed the tag for stable.
o Removed logit which got spilled from other patch to prevent compile failure.

Thanks,
Himanshu

---
 drivers/scsi/qla2xxx/qla_isr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/scsi/qla2xxx/qla_isr.c +3 -0
index e7bad0bfffda..4caec94d8e99 100644
@@ -1939,6 +1939,16 @@ static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req,
 		inbuf = (uint32_t *)&sts->nvme_ersp_data;
 		outbuf = (uint32_t *)fd->rspaddr;
 		iocb->u.nvme.rsp_pyld_len = le16_to_cpu(sts->nvme_rsp_pyld_len);
+		if (unlikely(iocb->u.nvme.rsp_pyld_len >
+		    sizeof(struct nvme_fc_ersp_iu))) {
+			WARN_ONCE(1, "Unexpected response payload length %u.\n",

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1954

Quoted 10 lines
drivers/scsi/qla2xxx/qla_isr.c +7 -0
+			    iocb->u.nvme.rsp_pyld_len);
+			ql_log(ql_log_warn, fcport->vha, 0x5100,
+			    "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);
+			iocb->u.nvme.rsp_pyld_len =
+			    sizeof(struct nvme_fc_ersp_iu);
+		}
 		iter = iocb->u.nvme.rsp_pyld_len >> 2;
 		for (; iter; iter--)
 			*outbuf++ = swab32(*inbuf++);

Thanks.

#2 lore raw
In reply to Himanshu Madhani

Himanshu,

Quoted 5 lines

On certain cases when response length is less than 32, NVME response data is supplied inline in IOCB. This is indicated by some combination of state flags. There was an instance when a high, and incorrect, response length was indicated causing driver to overrun buffers. Fix this by checking and limiting the response payload length.

Applied to 5.5/scsi-fixes, thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#3 lore raw
In reply to Himanshu Madhani
Quoted 7 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Himanshu Madhani
Sent: Tuesday, January 21, 2020 1:27 PM
Subject: [PATCH v4] qla2xxx: Fix unbound NVME response length

...

Quoted 2 lines

We discovered issue with our newer Gen7 adapter when response length happens to be larger than 32 bytes, could result into crash.

...

Quoted 4 lines
 drivers/scsi/qla2xxx/qla_isr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
(quoted diff) +0 -0

...

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1945

Quoted 4 lines
drivers/scsi/qla2xxx/qla_isr.c +4 -0
+		if (unlikely(iocb->u.nvme.rsp_pyld_len >
+		    sizeof(struct nvme_fc_ersp_iu))) {
+			WARN_ONCE(1, "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);

Do you really need a kernel stack dump for this error, which the WARN macros create? The problem would be caused by firmware behavior, not something wrong in the kernel.

If this function runs in interrupt context (based on the filename), then printing lots of data to the slow serial port can cause soft lockups and other issues.

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1951

Quoted 6 lines
drivers/scsi/qla2xxx/qla_isr.c +6 -0
+			ql_log(ql_log_warn, fcport->vha, 0x5100,
+			    "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);
+			iocb->u.nvme.rsp_pyld_len =
+			    sizeof(struct nvme_fc_ersp_iu);
+		}

If the problem is due to some firmware incompatibility and every response is long, the kernel log will quickly become full of these messages - per-IO prints are noisy. The handling implies the driver thinks it's safe to proceed, so there's nothing that is going to keep the problem from reoccurring. If the handling was to report a failed IO and shut down the device, then the number of possible error messages would quickly cease.

Safer approaches would be to print only once and maintain a count of errors in sysfs, or use ratelimited print functions.

#4 lore raw

Thanks for the review, Robert. Response inline..

Quoted 16 lines · Robert (Servers)
Quoted 7 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Himanshu Madhani
Sent: Tuesday, January 21, 2020 1:27 PM
Subject: [PATCH v4] qla2xxx: Fix unbound NVME response length

...

Quoted 2 lines

We discovered issue with our newer Gen7 adapter when response length happens to be larger than 32 bytes, could result into crash.

...

Quoted 4 lines
 drivers/scsi/qla2xxx/qla_isr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
(quoted diff) +0 -0

...

Quoted 8 lines

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1945

Quoted 4 lines
drivers/scsi/qla2xxx/qla_isr.c +4 -0
+		if (unlikely(iocb->u.nvme.rsp_pyld_len >
+		    sizeof(struct nvme_fc_ersp_iu))) {
+			WARN_ONCE(1, "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);

Do you really need a kernel stack dump for this error, which the WARN macros create? The problem would be caused by firmware behavior, not something wrong in the kernel.

The intent was to bring this to the tester's notice. My expectation is that this would be removed once the root cause is known. The issue was not reproducible internally.

Quoted 4 lines

If this function runs in interrupt context (based on the filename), then printing lots of data to the slow serial port can cause soft lockups and other issues.

In retrospect, this should have been under the driver debug tunable (which is set usually by testers).

Quoted 16 lines

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1951

Quoted 7 lines
drivers/scsi/qla2xxx/qla_isr.c +6 -0
+			ql_log(ql_log_warn, fcport->vha, 0x5100,
+			    "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);
+			iocb->u.nvme.rsp_pyld_len =
+			    sizeof(struct nvme_fc_ersp_iu);
+		}

If the problem is due to some firmware incompatibility and every response is long, the kernel log will quickly become full of these messages - per-IO prints are noisy. The handling implies the driver thinks it's safe to proceed, so there's nothing that is going to keep the problem from reoccurring. If the handling was to report a failed IO and shut down the device, then the number of possible error messages would quickly cease.

Quoted 2 lines

Safer approaches would be to print only once and maintain a count of errors in sysfs, or use ratelimited print functions.

I can post a follow on patch, with the WARN/log message under driver debug.

Regards,
-Arun

Quoted 3 lines
#5 lore raw

Arun,

Quoted 2 lines

I can post a follow on patch, with the WARN/log message under driver debug.

Just send a v5. Thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#6 lore raw
Quoted 7 lines · Martin K. Petersen

Arun,

Quoted 2 lines

I can post a follow on patch, with the WARN/log message under driver debug.

Just send a v5. Thanks!

Will do. Thanks Martin.

Regards,
-Arun

#7 lore raw
In reply to Arun Easi
Quoted 28 lines · Arun Easi

Thanks for the review, Robert. Response inline..

Quoted 15 lines · Robert (Servers)
Quoted 6 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Himanshu Madhani
Sent: Tuesday, January 21, 2020 1:27 PM
Subject: [PATCH v4] qla2xxx: Fix unbound NVME response length

...

Quoted 2 lines

We discovered issue with our newer Gen7 adapter when response length happens to be larger than 32 bytes, could result into crash.

...

Quoted 4 lines
 drivers/scsi/qla2xxx/qla_isr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
(quoted diff) +0 -0

...

Quoted 8 lines

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1945

Quoted 4 lines
drivers/scsi/qla2xxx/qla_isr.c +4 -0
+		if (unlikely(iocb->u.nvme.rsp_pyld_len >
+		    sizeof(struct nvme_fc_ersp_iu))) {
+			WARN_ONCE(1, "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);

Do you really need a kernel stack dump for this error, which the WARN macros create? The problem would be caused by firmware behavior, not something wrong in the kernel.

Quoted 3 lines

The intent was to bring this to the tester's notice. My expectation is that this would be removed once the root cause is known. The issue was not reproducible internally.

We have a reproducible test case, so testing the patch was not a problem. I agree the log message should be restricted or suppressed though, once would be enough.

The problem appeared to be that an extra bit was set (because the length was too long by 16K) and our testing worked OK with the ersp_iu data truncated to the correct structure size.

-Ewan

Quoted 34 lines
Quoted 4 lines

If this function runs in interrupt context (based on the filename), then printing lots of data to the slow serial port can cause soft lockups and other issues.

In retrospect, this should have been under the driver debug tunable (which is set usually by testers).

Quoted 15 lines

drivers/scsi/qla2xxx/qla_isr.c static void qla24xx_nvme_iocb_entry(scsi_qla_host_t *vha, struct req_que *req, · line 1951

Quoted 6 lines
drivers/scsi/qla2xxx/qla_isr.c +6 -0
+			ql_log(ql_log_warn, fcport->vha, 0x5100,
+			    "Unexpected response payload length %u.\n",
+			    iocb->u.nvme.rsp_pyld_len);
+			iocb->u.nvme.rsp_pyld_len =
+			    sizeof(struct nvme_fc_ersp_iu);
+		}

If the problem is due to some firmware incompatibility and every response is long, the kernel log will quickly become full of these messages - per-IO prints are noisy. The handling implies the driver thinks it's safe to proceed, so there's nothing that is going to keep the problem from reoccurring. If the handling was to report a failed IO and shut down the device, then the number of possible error messages would quickly cease.

Quoted 2 lines

Safer approaches would be to print only once and maintain a count of errors in sysfs, or use ratelimited print functions.

I can post a follow on patch, with the WARN/log message under driver debug.

Regards,
-Arun

Quoted 2 lines