target: pscsi: avoid OOM in pscsi_map_sg()
Subsystems (2)
PATCH 1/1 target: pscsi: avoid OOM in pscsi_map_sg()
From: Martin Wilck <[email protected]>
pscsi_map_sg() uses the variable nr_pages as a hint for bio_kmalloc() how many vector elements to allocate. If nr_pages is < BIO_MAX_PAGES, it will be reset to 0 after successful allocation of the bio.
If bio_add_pc_page() fails later for whatever reason, pscsi_map_sg() tries to allocate another bio, passing nr_vecs=0. This causes bio_add_pc_page() to fail immediately in the next call. pci_map_sg() continues to allocate zero-length bios until memory is exhausted and the kernel crashes with OOM. This can be easily observed by exporting a SATA DVD drive via pscsi. The target crashes as soon as the client tries to access the DVD LUN. In the case I analyzed, bio_add_pc_page() would fail because the DVD device's max_sectors_kb (128) was exceeded.
Avoid this by simply not resetting nr_pages to 0 after allocating the bio. This way, the client receives an IO error when it tries to send requests exceeding the devices max_sectors_kb, and eventually gets it right. The client must still limit max_sectors_kb e.g. by an udev rule if (like in my case) the driver doesn't report valid block limits, otherwise it encounters I/O errors.
--- drivers/target/target_core_pscsi.c | 1 - 1 file changed, 1 deletion(-)
drivers/target/target_core_pscsi.c +0 -1
@@ -881,7 +881,6 @@ pscsi_map_sg(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents,if (!bio) { new_bio: nr_vecs =bio_max_segs (nr_pages); - nr_pages -= nr_vecs;/* * Calls bio_kmalloc() and sets bio->bi_end_io() */
Signature · 1 line
2.26.2
Quoted 6 lines · On Tue, Mar 23, 2021 at 05:22:03PM +0100, [email protected] wrote:
Avoid this by simply not resetting
nr_pagesto 0 after allocating the bio. This way, the client receives an IO error when it tries to send requests exceeding the devicesmax_sectors_kb, and eventually gets it right. The client must still limitmax_sectors_kbe.g. by an udev rule if (like in my case) the driver doesn't report valid block limits, otherwise it encounters I/O errors.
FYI, I think the what you did here is correct, but not enough. When pscsi_get_bio (that is bio_kmalloc) fails, this function needs to unwind and return an error insted of blindly retrying the allocation, else we can't recover from a memory shortage.
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · On Tue, Mar 23, 2021 at 05:22:03PM +0100, [email protected] wrote:
Avoid this by simply not resetting
nr_pagesto 0 after allocating the bio. This way, the client receives an IO error when it tries to send requests exceeding the devicesmax_sectors_kb, and eventually gets it right. The client must still limitmax_sectors_kbe.g. by an udev rule if (like in my case) the driver doesn't report valid block limits, otherwise it encounters I/O errors.
Quoted 5 lines
FYI, I think the what you did here is correct, but not enough. When
pscsi_get_bio(that isbio_kmalloc) fails, this function needs to unwind and return an error insted of blindly retrying the allocation,
else we can't recover from a memory shortage.
I can try to do that, but in the tests I ran, I never observed bio_kmalloc() failing. I saw it eat all memory, and various processes being killed by the OOM killer, before the system eventually panicked with OOM. It takes only fractions of a second until this happens:
[ 57.356238] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=lightdm-gtk-gre,pid=4586,uid=481 [ 57.369635] Out of memory: Killed process 4586 (lightdm-gtk-gre) total-vm:1035752kB, anon-rss:0kB, file-rss:2936kB, shmem-rss:0kB ... [ 57.698656] Node 0 Normal free:54828kB min:55008kB low:68760kB high:82512kB active_anon:4kB inactive_anon:0kB active_file:936kB inactive_file:1164kB unevictable:58036kB writepending:720kB present:13631488kB managed:13310252kB mlocked:58036kB kernel_stack:5808kB pagetables:8972kB bounce:0kB free_pcp:564kB local_pcp:0kB free_cma:0kB ... [ 57.818978] Unreclaimable slab info: [ 58.254160] kmalloc-192 15683546KB 15683546KB
(system has 16GiB memory).
Martin
Martin,
Quoted 5 lines · On 3/23/21 09:23, [email protected] wrote:
From: Martin Wilck <[email protected]>
pscsi_map_sg()uses the variablenr_pagesas a hint forbio_kmalloc()how many vector elements to allocate. Ifnr_pagesis <BIO_MAX_PAGES, it will be reset to 0 after successful allocation of the bio.
I think BIO_MAX_PAGES is replaced by BIO_MAX_VECS with commit a8affc03a9b3 ("block: rename BIO_MAX_PAGES to BIO_MAX_VECS").
Quoted 5 lines · Chaitanya Kulkarni
Martin,
Quoted 2 lines · On 3/23/21 09:23, [email protected] wrote:
From: Martin Wilck <[email protected]>
Quoted 8 lines
Quoted 5 lines
pscsi_map_sg()uses the variablenr_pagesas a hint forbio_kmalloc()
how many vector elements to allocate. Ifnr_pagesis <BIO_MAX_PAGES,
it will be reset to 0 after successful allocation of the bio.I think
BIO_MAX_PAGESis replaced byBIO_MAX_VECSwith commita8affc03a9b3("block: renameBIO_MAX_PAGEStoBIO_MAX_VECS").
Right. I made my patch against mkp/queue, which doesn't include this commit yet. As this is just in the description, I don't think it matters much, does it?
Martin
Quoted 10 lines · Martin Wilck
Quoted 2 lines
I think
BIO_MAX_PAGESis replaced byBIO_MAX_VECSwith commita8affc03a9b3("block: renameBIO_MAX_PAGEStoBIO_MAX_VECS").Right. I made my patch against mkp/queue, which doesn't include this commit yet. As this is just in the description, I don't think it matters much, does it?
Martin
I don't think it does, I'll let you decide that.