Skip to contentSkip to conversation

vhost-scsi: Prevent OOM from invalid protection SGL count

PATCH In review Jia Jia 4 participants 5 messages Jul 17 – Jul 18

Subsystems (7)
lore
Share & tools
lore

Jia Jia

author
#0 lore raw

PATCH 1/1 vhost-scsi: Prevent OOM from invalid protection SGL count

The protection SGL path passes the result of vhost_scsi_calc_sgls() directly to sg_alloc_table_chained(). The helper returns a negative errno when the iterator is invalid or the request exceeds the segment limit. The negative errno is then treated as a very large unsigned count and sends the request into the SGL allocation path with an invalid size.

Repeated malformed T10-PI submissions from a host-side application caused memory usage to rise sharply. MemAvailable fell to about 200 MB, and PSI full avg10 reached about 1.46. The OOM killer terminated several userspace processes before the endpoint cleanup completed. The kernel log included:

    [17036.451028] Out of memory: Killed process 2345 (systemd)
    [17036.493325] Out of memory: Killed process 2349 (sd-pam)
    [17078.265127] Out of memory: Killed process 1793 (networkd-dispat)

Return the calculation error before setting up the protection SGL. This keeps the protection path consistent with the data SGL path and prevents the invalid count from entering the allocation path.

Fixes bca939d5bcd0 ("vhost-scsi: Dynamically allocate scatterlists")
---
 drivers/vhost/scsi.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/vhost/scsi.c +3 -0
index 9a1253b9d..8486652fd 100644
@@ -972,6 +972,9 @@ vhost_scsi_mapal(struct vhost_scsi *vs, struct vhost_scsi_cmd *cmd,
 	if (prot_bytes) {
 		sgl_count = vhost_scsi_calc_sgls(prot_iter, prot_bytes,
 						 VHOST_SCSI_PREALLOC_PROT_SGLS);
+		if (sgl_count < 0)
+			return sgl_count;
+
 		cmd->prot_table.sgl = cmd->prot_sgl;
 		ret = sg_alloc_table_chained(&cmd->prot_table, sgl_count,
 					     cmd->prot_table.sgl,
Signature · 1 line
2.43.0
maintainer
#1 lore raw

cc'ing Linfeng.

Linfeng also reported this bug and provided a patch that has your check and an extra check to avoid calling into the function in the first place. It's been stuck in some other list.

I think it might be best for Linfeng to re-submit their patch now so it can get merged. For that patch, I think there was one outstanding question left where Michael had asked if the issue was found with AI so the proper tag could be added. I don't think that question was responded to.

Quoted 43 lines · Jia Jia

The protection SGL path passes the result of vhost_scsi_calc_sgls() directly to sg_alloc_table_chained(). The helper returns a negative errno when the iterator is invalid or the request exceeds the segment limit. The negative errno is then treated as a very large unsigned count and sends the request into the SGL allocation path with an invalid size.

Repeated malformed T10-PI submissions from a host-side application caused memory usage to rise sharply. MemAvailable fell to about 200 MB, and PSI full avg10 reached about 1.46. The OOM killer terminated several userspace processes before the endpoint cleanup completed. The kernel log included:

     [17036.451028] Out of memory: Killed process 2345 (systemd)
     [17036.493325] Out of memory: Killed process 2349 (sd-pam)
     [17078.265127] Out of memory: Killed process 1793 (networkd-dispat)

Return the calculation error before setting up the protection SGL. This keeps the protection path consistent with the data SGL path and prevents the invalid count from entering the allocation path.

Fixes bca939d5bcd0 ("vhost-scsi: Dynamically allocate scatterlists")
---
  drivers/vhost/scsi.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/vhost/scsi.c +3 -0
index 9a1253b9d..8486652fd 100644
@@ -972,6 +972,9 @@ vhost_scsi_mapal(struct vhost_scsi *vs, struct vhost_scsi_cmd *cmd,
  	if (prot_bytes) {
  		sgl_count = vhost_scsi_calc_sgls(prot_iter, prot_bytes,
  						 VHOST_SCSI_PREALLOC_PROT_SGLS);
+		if (sgl_count < 0)
+			return sgl_count;
+
  		cmd->prot_table.sgl = cmd->prot_sgl;
  		ret = sg_alloc_table_chained(&cmd->prot_table, sgl_count,
  					     cmd->prot_table.sgl,
Signature · 1 line
2.43.0
#2 lore raw
Quoted 10 lines · Mike Christie

cc'ing Linfeng.

Linfeng also reported this bug and provided a patch that has your check and an extra check to avoid calling into the function in the first place. It's been stuck in some other list.

I think it might be best for Linfeng to re-submit their patch now so it can get merged. For that patch, I think there was one outstanding question left where Michael had asked if the issue was found with AI so the proper tag could be added. I don't think that question was responded to.

Right, was waiting for that.

Quoted 45 lines · Jia Jia
Quoted 43 lines

The protection SGL path passes the result of vhost_scsi_calc_sgls() directly to sg_alloc_table_chained(). The helper returns a negative errno when the iterator is invalid or the request exceeds the segment limit. The negative errno is then treated as a very large unsigned count and sends the request into the SGL allocation path with an invalid size.

Repeated malformed T10-PI submissions from a host-side application caused memory usage to rise sharply. MemAvailable fell to about 200 MB, and PSI full avg10 reached about 1.46. The OOM killer terminated several userspace processes before the endpoint cleanup completed. The kernel log included:

     [17036.451028] Out of memory: Killed process 2345 (systemd)
     [17036.493325] Out of memory: Killed process 2349 (sd-pam)
     [17078.265127] Out of memory: Killed process 1793 (networkd-dispat)

Return the calculation error before setting up the protection SGL. This keeps the protection path consistent with the data SGL path and prevents the invalid count from entering the allocation path.

Fixes bca939d5bcd0 ("vhost-scsi: Dynamically allocate scatterlists")
---
  drivers/vhost/scsi.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/vhost/scsi.c +3 -0
index 9a1253b9d..8486652fd 100644
@@ -972,6 +972,9 @@ vhost_scsi_mapal(struct vhost_scsi *vs, struct vhost_scsi_cmd *cmd,
  	if (prot_bytes) {
  		sgl_count = vhost_scsi_calc_sgls(prot_iter, prot_bytes,
  						 VHOST_SCSI_PREALLOC_PROT_SGLS);
+		if (sgl_count < 0)
+			return sgl_count;
+
  		cmd->prot_table.sgl = cmd->prot_sgl;
  		ret = sg_alloc_table_chained(&cmd->prot_table, sgl_count,
  					     cmd->prot_table.sgl,
Signature · 1 line
2.43.0
author
#3 lore raw
In reply to Mike Christie

Before sending my patch, I did a duplicate check against the upstream linux-next tree, upstream history, publicly searchable mailing-list archives, and Patchwork. I searched using the vhost-scsi and scatterlist function names, T10-PI, invalid protection SGL counts, and the OOM/resource-exhaustion symptoms. I did not find Linfeng's earlier patch.

Could you please point me to Linfeng's patch, or confirm what the additional check covers? In particular, does it also validate:

prot_bytes <= exp_data_len

before subtracting prot_bytes from exp_data_len and entering the SGL mapping path? I had identified this as a related PI length-mismatch path, but did not include it in the patch I sent.

With a one-byte combined payload and pi_bytesout=2, KGDB stopped at the following call stack:

    #0  sg_alloc_table_chained(
            table=0xffff88810bbc5108,
            nents=0,
            first_chunk=0xffff888110050000,
            nents_first_chunk=2048)
        at lib/sg_pool.c:117

    #1  vhost_scsi_handle_vq+2295
    #2  vhost worker task
    #3  srso_alias_return_thunk

The stop was at the BUG_ON(!nents) instruction in sg_alloc_table_chained(), and the kernel log recorded kernel BUG at lib/sg_pool.c:117!. This confirms that the PI length check is needed before the iterator adjustment and SGL mapping path.

If Linfeng's patch covers both the invalid protection SGL count and this PI length check, I will not submit duplicate work.

Regarding the AI question: AI assistance was used during the source analysis and test development. I also checked the relevant code path and the host-side behavior.

#4 lore raw
In reply to Jia Jia
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] A malicious guest can reliably crash the host kernel by providing a virtio-scsi request where the T10 Protection Information length equals or exceeds the total data payload size, triggering a BUG_ON() during SGL allocation. --

--- Patch [1]: [PATCH] vhost-scsi: Prevent OOM from invalid protection SGL count --- commit ee69ef365d50119dd5831e263cc68413e6e2bee1
Author: Jia Jia <[email protected]>

vhost-scsi: Prevent OOM from invalid protection SGL count

The protection SGL path passes the result of vhost_scsi_calc_sgls() directly to sg_alloc_table_chained(). The helper returns a negative errno when the iterator is invalid or the request exceeds the segment limit. The negative errno is then treated as a very large unsigned count and sends the request into the SGL allocation path with an invalid size.

drivers/vhost/scsi.c vhost_scsi_mapal(struct vhost_scsi *vs, struct vhost_scsi_cmd *cmd, · line 977

Quoted 11 lines
drivers/vhost/scsi.c +3 -0
index 9a1253b9d8c50..8486652fd31e8 100644
@@ -972,6 +972,9 @@ vhost_scsi_mapal(struct vhost_scsi *vs, struct vhost_scsi_cmd *cmd,
 	if (prot_bytes) {
 		sgl_count = vhost_scsi_calc_sgls(prot_iter, prot_bytes,
 						 VHOST_SCSI_PREALLOC_PROT_SGLS);
+		if (sgl_count < 0)
+			return sgl_count;
+

[Severity: Critical]
This is a pre-existing issue, but could sgl_count evaluate to exactly 0 here?

If a guest provides a virtio-scsi request where the protection information length (prot_bytes) equals or exceeds the total data payload size (exp_data_len), vhost_scsi_handle_vq() subtracts it without checking for underflow or equality:

drivers/vhost/scsi.c:vhost_scsi_handle_vq() {
        ...
        if (prot_bytes) {
                exp_data_len -= prot_bytes;
                prot_iter = data_iter;
                iov_iter_truncate(&prot_iter, prot_bytes);
                iov_iter_advance(&data_iter, prot_bytes);
        }
        ...
}

This can reduce the data iterator's count to 0. When vhost_scsi_mapal() calls vhost_scsi_calc_sgls() on the remaining iterators, iov_iter_npages() evaluates to 0, returning an sgl_count of 0.

Because the error check added here only tests for < 0, a 0 value will slip through and be passed as nents to sg_alloc_table_chained(), which will panic the host kernel:

lib/sg_pool.c:sg_alloc_table_chained() {
        ...
        BUG_ON(!nents);
        ...
}

Would it make sense to change this check to if (sgl_count <= 0) (and similarly for the data SGL check below it) to prevent a guest-triggerable host panic?

Quoted 3 lines
drivers/vhost/scsi.c +0 -0
 		cmd->prot_table.sgl = cmd->prot_sgl;
 		ret = sg_alloc_table_chained(&cmd->prot_table, sgl_count,
 					     cmd->prot_table.sgl,
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1