ceph: fix OOB read in ceph_osdc_list_watchers via uncapped outdata_len
Subsystems (2)
PATCH 1/1 ceph: fix OOB read in ceph_osdc_list_watchers via uncapped outdata_len
The OSD reply header field op->payload_len is wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:
m->outdata_len[i] = le32_to_cpu(op->payload_len);
This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in ceph_osdc_list_watchers():
void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO) ceph_osd_data_pages_init(..., PAGE_SIZE, ...)
The messenger caps the copy to PAGE_SIZE bytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing _safe decoder boundary checks to pass while the physical reads cross the slab allocation boundary.
KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
Read of size 4 at addr ffff88800a229f9e by task insmod/57
CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy)
Tainted: [O]=OOT_MODULE
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4d/0x70
print_report+0x170/0x4f3
? __pfx__raw_spin_lock_irqsave+0x10/0x10
kasan_report+0xda/0x110
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc]
ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
? __pfx_do_one_initcall+0x10/0x10
? kasan_unpoison+0x44/0x70
do_init_module+0x27c/0x790
? __pfx_do_init_module+0x10/0x10
? __kasan_slab_free+0x47/0x70
? kfree+0x15f/0x3b0
load_module+0x4a9a/0x6350
? __pfx_load_module+0x10/0x10
? security_file_permission+0x24/0x50
? kernel_read_file+0x2ed/0x770
? init_module_from_file+0x15c/0x180
init_module_from_file+0x15c/0x180
? __pfx_init_module_from_file+0x10/0x10
? tick_nohz_handler+0x2a3/0x640
? _raw_spin_lock+0x7e/0xd0
idempotent_init_module+0x21f/0x750
? __pfx_idempotent_init_module+0x10/0x10
? fdget+0x4e/0x4a0
? fdget+0x4e/0x4a0
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
? exc_page_fault+0x66/0xb0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Allocated by task 57:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_kmalloc+0x7f/0x90
ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
do_init_module+0x27c/0x790
load_module+0x4a9a/0x6350
init_module_from_file+0x15c/0x180
idempotent_init_module+0x21f/0x750
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88800a229000
which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 3998 bytes inside of
allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0)
Memory state around the buggy address:
ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by capping the decode window end to PAGE_SIZE, matching the actual allocation size.
Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing CEPH_OSD_OP_LIST_WATCHERS without further privileges beyond OSD session establishment.
--- net/ceph/osd_client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/osd_client.c +1 -1
@@ -5091,7 +5091,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p + req->r_ops[0 ].outdata_len; +void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len,PAGE_SIZE ); ret =decode_watchers (&p, end, watchers, num_watchers); }
Signature · 1 line
2.53.0
Quoted 112 lines · Pavitra Jha
The OSD reply header field
op->payload_lenis wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:m->outdata_len[i] = le32_to_cpu(op->payload_len);This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in
ceph_osdc_list_watchers():void *const end = p + req->r_ops[0].outdata_len;The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO)ceph_osd_data_pages_init(..., PAGE_SIZE, ...)The messenger caps the copy to
PAGE_SIZEbytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing_safedecoder boundary checks to pass while the physical reads cross the slab allocation boundary.KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] Read of size 4 at addr ffff88800a229f9e by task insmod/57 CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy) Tainted: [O]=OOT_MODULE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4d/0x70 print_report+0x170/0x4f3 ? __pfx__raw_spin_lock_irqsave+0x10/0x10 kasan_report+0xda/0x110 ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc] ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 ? __pfx_do_one_initcall+0x10/0x10 ? kasan_unpoison+0x44/0x70 do_init_module+0x27c/0x790 ? __pfx_do_init_module+0x10/0x10 ? __kasan_slab_free+0x47/0x70 ? kfree+0x15f/0x3b0 load_module+0x4a9a/0x6350 ? __pfx_load_module+0x10/0x10 ? security_file_permission+0x24/0x50 ? kernel_read_file+0x2ed/0x770 ? init_module_from_file+0x15c/0x180 init_module_from_file+0x15c/0x180 ? __pfx_init_module_from_file+0x10/0x10 ? tick_nohz_handler+0x2a3/0x640 ? _raw_spin_lock+0x7e/0xd0 idempotent_init_module+0x21f/0x750 ? __pfx_idempotent_init_module+0x10/0x10 ? fdget+0x4e/0x4a0 ? fdget+0x4e/0x4a0 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 ? exc_page_fault+0x66/0xb0 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 57: kasan_save_stack+0x30/0x50 kasan_save_track+0x14/0x30 __kasan_kmalloc+0x7f/0x90 ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 do_init_module+0x27c/0x790 load_module+0x4a9a/0x6350 init_module_from_file+0x15c/0x180 idempotent_init_module+0x21f/0x750 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 entry_SYSCALL_64_after_hwframe+0x77/0x7f The buggy address belongs to the object at ffff88800a229000 which belongs to the cache kmalloc-4k of size 4096 The buggy address is located 3998 bytes inside of allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0) Memory state around the buggy address: ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc ^ ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by capping the decode window end to
PAGE_SIZE, matching the actual allocation size.Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing
CEPH_OSD_OP_LIST_WATCHERSwithout further privileges beyond OSD session establishment.Fixes a4ed38d7a180 ("libceph: support for CEPH_OSD_OP_LIST_WATCHERS")Signed-off-by Pavitra Jha <[email protected]>--- net/ceph/osd_client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)net/ceph/osd_client.c +0
-0
net/ceph/osd_client.c int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, · line 5094
Quoted 5 lines
net/ceph/osd_client.c +1
-1@@ -5091,7 +5091,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p + req->r_ops[0 ].outdata_len; +void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len,PAGE_SIZE );
We cannot do assumption about the buffer size. Currently, it looks like a hardcoded value. Potentially, the buffer size could change and it could be the bug reason.
We need to introduce a variable that should keep PAGE_SIZE value and it can be used in ceph_osd_data_pages_init(..., PAGE_SIZE, ...) and in min_t(u32, req-
Quoted 1 line
r_ops[0].outdata_len,
PAGE_SIZE).
Thanks,
Slava.
Quoted 3 lines
net/ceph/osd_client.c +0
-0ret =decode_watchers (&p, end, watchers, num_watchers); }
PATCH v2 1/1 ceph: fix OOB read in ceph_osdc_list_watchers via uncapped outdata_len
The OSD reply header field op->payload_len is wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:
m->outdata_len[i] = le32_to_cpu(op->payload_len);
This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in ceph_osdc_list_watchers():
void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO) ceph_osd_data_pages_init(..., PAGE_SIZE, ...)
The messenger caps the copy to PAGE_SIZE bytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing _safe decoder boundary checks to pass while the physical reads cross the slab allocation boundary.
KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
Read of size 4 at addr ffff88800a229f9e by task insmod/57
CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy)
Tainted: [O]=OOT_MODULE
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4d/0x70
print_report+0x170/0x4f3
? __pfx__raw_spin_lock_irqsave+0x10/0x10
kasan_report+0xda/0x110
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc]
ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
? __pfx_do_one_initcall+0x10/0x10
? kasan_unpoison+0x44/0x70
do_init_module+0x27c/0x790
? __pfx_do_init_module+0x10/0x10
? __kasan_slab_free+0x47/0x70
? kfree+0x15f/0x3b0
load_module+0x4a9a/0x6350
? __pfx_load_module+0x10/0x10
? security_file_permission+0x24/0x50
? kernel_read_file+0x2ed/0x770
? init_module_from_file+0x15c/0x180
init_module_from_file+0x15c/0x180
? __pfx_init_module_from_file+0x10/0x10
? tick_nohz_handler+0x2a3/0x640
? _raw_spin_lock+0x7e/0xd0
idempotent_init_module+0x21f/0x750
? __pfx_idempotent_init_module+0x10/0x10
? fdget+0x4e/0x4a0
? fdget+0x4e/0x4a0
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
? exc_page_fault+0x66/0xb0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Allocated by task 57:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_kmalloc+0x7f/0x90
ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
do_init_module+0x27c/0x790
load_module+0x4a9a/0x6350
init_module_from_file+0x15c/0x180
idempotent_init_module+0x21f/0x750
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
entry_YSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88800a229000
which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 3998 bytes inside of
allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0)
Memory state around the buggy address:
ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing buf_len to hold the allocation size, using it in both ceph_osd_data_pages_init() and the min_t() decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes.
Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing CEPH_OSD_OP_LIST_WATCHERS without further privileges beyond OSD session establishment.
---
v2: Introduce buf_len variable instead of hardcoding PAGE_SIZE
independently in ceph_osd_data_pages_init() and the min_t() cap, per Viacheslav Dubeyko's review.
--- net/ceph/osd_client.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +3 -2
@@ -5063,6 +5063,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_osd_request *req;struct page **pages;int ret; +const size_t buf_len =PAGE_SIZE ; req =ceph_osdc_alloc_request (osdc,NULL ,1 , false,GFP_NOIO );if (!req)
@@ -5081,7 +5082,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,osd_req_op_init (req,0 ,CEPH_OSD_OP_LIST_WATCHERS ,0 );ceph_osd_data_pages_init (osd_req_op_data (req,0 , list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true); ret =ceph_osdc_alloc_messages (req,GFP_NOIO );if (ret)
@@ -5091,7 +5092,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len,PAGE_SIZE ); +void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len, buf_len); ret =decode_watchers (&p, end, watchers, num_watchers); }
Signature · 1 line
2.53.0
Quoted 135 lines · Pavitra Jha
The OSD reply header field
op->payload_lenis wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:m->outdata_len[i] = le32_to_cpu(op->payload_len);This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in
ceph_osdc_list_watchers():void *const end = p + req->r_ops[0].outdata_len;The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO)ceph_osd_data_pages_init(..., PAGE_SIZE, ...)The messenger caps the copy to
PAGE_SIZEbytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing_safedecoder boundary checks to pass while the physical reads cross the slab allocation boundary.KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] Read of size 4 at addr ffff88800a229f9e by task insmod/57 CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy) Tainted: [O]=OOT_MODULE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4d/0x70 print_report+0x170/0x4f3 ? __pfx__raw_spin_lock_irqsave+0x10/0x10 kasan_report+0xda/0x110 ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc] ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 ? __pfx_do_one_initcall+0x10/0x10 ? kasan_unpoison+0x44/0x70 do_init_module+0x27c/0x790 ? __pfx_do_init_module+0x10/0x10 ? __kasan_slab_free+0x47/0x70 ? kfree+0x15f/0x3b0 load_module+0x4a9a/0x6350 ? __pfx_load_module+0x10/0x10 ? security_file_permission+0x24/0x50 ? kernel_read_file+0x2ed/0x770 ? init_module_from_file+0x15c/0x180 init_module_from_file+0x15c/0x180 ? __pfx_init_module_from_file+0x10/0x10 ? tick_nohz_handler+0x2a3/0x640 ? _raw_spin_lock+0x7e/0xd0 idempotent_init_module+0x21f/0x750 ? __pfx_idempotent_init_module+0x10/0x10 ? fdget+0x4e/0x4a0 ? fdget+0x4e/0x4a0 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 ? exc_page_fault+0x66/0xb0 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 57: kasan_save_stack+0x30/0x50 kasan_save_track+0x14/0x30 __kasan_kmalloc+0x7f/0x90 ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 do_init_module+0x27c/0x790 load_module+0x4a9a/0x6350 init_module_from_file+0x15c/0x180 idempotent_init_module+0x21f/0x750 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 entry_YSCALL_64_after_hwframe+0x77/0x7f The buggy address belongs to the object at ffff88800a229000 which belongs to the cache kmalloc-4k of size 4096 The buggy address is located 3998 bytes inside of allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0) Memory state around the buggy address: ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc ^ ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing
buf_lento hold the allocation size, using it in bothceph_osd_data_pages_init()and themin_t()decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes.Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing
CEPH_OSD_OP_LIST_WATCHERSwithout further privileges beyond OSD session establishment.Fixes a4ed38d7a180 ("libceph: support for CEPH_OSD_OP_LIST_WATCHERS")Signed-off-by Pavitra Jha <[email protected]>---
v2: Introducebuf_lenvariable instead of hardcodingPAGE_SIZE
independently inceph_osd_data_pages_init()and themin_t()cap, per Viacheslav Dubeyko's review.--- net/ceph/osd_client.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)net/ceph/osd_client.c +2
-1@@ -5063,6 +5063,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_osd_request *req;struct page **pages;int ret; +const size_t buf_len =PAGE_SIZE ; req =ceph_osdc_alloc_request (osdc,NULL ,1 , false,GFP_NOIO );if (!req)@@ -5081,7 +5082,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,osd_req_op_init (req,0 ,CEPH_OSD_OP_LIST_WATCHERS ,0 );ceph_osd_data_pages_init (osd_req_op_data (req,0 , list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true); ret =ceph_osdc_alloc_messages (req,GFP_NOIO );if (ret)
net/ceph/osd_client.c int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, · line 5095
Quoted 5 lines
net/ceph/osd_client.c +1
-1@@ -5091,7 +5092,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len,PAGE_SIZE ); +void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len, buf_len);
This patch is not correctly prepared because initially the patch has this:
- void *const end = p + req->r_ops[0].outdata_len;
+ void *const end = p + min_t(u32, req->r_ops[0].outdata_len,PAGE_SIZE);
Also, this line has 85 symbols. This statement should be reworked to be not longer than 80 symbols.
Thanks,
Slava.
Quoted 3 lines
net/ceph/osd_client.c +0
-0ret =decode_watchers (&p, end, watchers, num_watchers); }
PATCH v3 1/1 ceph: fix OOB read in ceph_osdc_list_watchers via uncapped outdata_len
The OSD reply header field op->payload_len is wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:
m->outdata_len[i] = le32_to_cpu(op->payload_len);
This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in ceph_osdc_list_watchers():
void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO) ceph_osd_data_pages_init(..., PAGE_SIZE, ...)
The messenger caps the copy to PAGE_SIZE bytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing _safe decoder boundary checks to pass while the physical reads cross the slab allocation boundary.
KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
Read of size 4 at addr ffff88800a229f9e by task insmod/57
CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy)
Tainted: [O]=OOT_MODULE
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4d/0x70
print_report+0x170/0x4f3
? __pfx__raw_spin_lock_irqsave+0x10/0x10
kasan_report+0xda/0x110
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc]
ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
? __pfx_do_one_initcall+0x10/0x10
? kasan_unpoison+0x44/0x70
do_init_module+0x27c/0x790
? __pfx_do_init_module+0x10/0x10
? __kasan_slab_free+0x47/0x70
? kfree+0x15f/0x3b0
load_module+0x4a9a/0x6350
? __pfx_load_module+0x10/0x10
? security_file_permission+0x24/0x50
? kernel_read_file+0x2ed/0x770
? init_module_from_file+0x15c/0x180
init_module_from_file+0x15c/0x180
? __pfx_init_module_from_file+0x10/0x10
? tick_nohz_handler+0x2a3/0x640
? _raw_spin_lock+0x7e/0xd0
idempotent_init_module+0x21f/0x750
? __pfx_idempotent_init_module+0x10/0x10
? fdget+0x4e/0x4a0
? fdget+0x4e/0x4a0
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
? exc_page_fault+0x66/0xb0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Allocated by task 57:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_kmalloc+0x7f/0x90
ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
do_init_module+0x27c/0x790
load_module+0x4a9a/0x6350
init_module_from_file+0x15c/0x180
idempotent_init_module+0x21f/0x750
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88800a229000
which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 3998 bytes inside of
allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0)
Memory state around the buggy address:
ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing buf_len to hold the allocation size, using it in both ceph_osd_data_pages_init() and the min_t() decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes.
Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing CEPH_OSD_OP_LIST_WATCHERS without further privileges beyond OSD session establishment.
---
v3: Split overlong min_t() line to fit 80-column limit,
per Viacheslav Dubeyko's review of v2.
v2: Introduce buf_len variable instead of hardcoding PAGE_SIZE
independently in ceph_osd_data_pages_init() and the min_t() cap, per Viacheslav Dubeyko's review.
--- net/ceph/osd_client.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +4 -2
@@ -5063,6 +5063,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_osd_request *req;struct page **pages;int ret; +const size_t buf_len =PAGE_SIZE ; req =ceph_osdc_alloc_request (osdc,NULL ,1 , false,GFP_NOIO );if (!req)
@@ -5081,7 +5082,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,osd_req_op_init (req,0 ,CEPH_OSD_OP_LIST_WATCHERS ,0 );ceph_osd_data_pages_init (osd_req_op_data (req,0 , list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true); ret =ceph_osdc_alloc_messages (req,GFP_NOIO );if (ret)
@@ -5091,7 +5092,8 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len,PAGE_SIZE ); +void *const end = p + +min_t (u32, req->r_ops[0 ].outdata_len, buf_len); ret =decode_watchers (&p, end, watchers, num_watchers); }
Signature · 1 line
2.53.0
Viacheslav Dubeyko
maintainerQuoted 145 lines · Pavitra Jha
The OSD reply header field
op->payload_lenis wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:m->outdata_len[i] =
le32_to_cpu(op->payload_len);This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in
ceph_osdc_list_watchers():void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO)ceph_osd_data_pages_init(..., PAGE_SIZE, ...)The messenger caps the copy to
PAGE_SIZEbytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing_safedecoder boundary checks to pass while the physical reads cross the slab allocation boundary.KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] Read of size 4 at addr ffff88800a229f9e by task insmod/57 CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy) Tainted: [O]=OOT_MODULE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0- debian-1.17.0-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4d/0x70 print_report+0x170/0x4f3 ? __pfx__raw_spin_lock_irqsave+0x10/0x10 kasan_report+0xda/0x110 ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc] ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 ? __pfx_do_one_initcall+0x10/0x10 ? kasan_unpoison+0x44/0x70 do_init_module+0x27c/0x790 ? __pfx_do_init_module+0x10/0x10 ? __kasan_slab_free+0x47/0x70 ? kfree+0x15f/0x3b0 load_module+0x4a9a/0x6350 ? __pfx_load_module+0x10/0x10 ? security_file_permission+0x24/0x50 ? kernel_read_file+0x2ed/0x770 ? init_module_from_file+0x15c/0x180 init_module_from_file+0x15c/0x180 ? __pfx_init_module_from_file+0x10/0x10 ? tick_nohz_handler+0x2a3/0x640 ? _raw_spin_lock+0x7e/0xd0 idempotent_init_module+0x21f/0x750 ? __pfx_idempotent_init_module+0x10/0x10 ? fdget+0x4e/0x4a0 ? fdget+0x4e/0x4a0 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 ? exc_page_fault+0x66/0xb0 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 57: kasan_save_stack+0x30/0x50 kasan_save_track+0x14/0x30 __kasan_kmalloc+0x7f/0x90 ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 do_init_module+0x27c/0x790 load_module+0x4a9a/0x6350 init_module_from_file+0x15c/0x180 idempotent_init_module+0x21f/0x750 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 entry_SYSCALL_64_after_hwframe+0x77/0x7f The buggy address belongs to the object at ffff88800a229000which belongs to the cache kmalloc-4k of size 4096 The buggy address is located 3998 bytes inside of allocated 4000-byte region [
ffff88800a229000,ffff88800a229fa0)Memory state around the buggy address: ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing
buf_lento hold the allocation size, using it in bothceph_osd_data_pages_init()and themin_t()decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes.Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing
CEPH_OSD_OP_LIST_WATCHERSwithout further privileges beyond OSD session establishment.Fixes a4ed38d7a180 ("libceph: support for
CEPH_OSD_OP_LIST_WATCHERS")Signed-off-by Pavitra Jha <[email protected]>---
v3: Split overlongmin_t()line to fit 80-column limit, per Viacheslav Dubeyko's review of v2.
v2: Introducebuf_lenvariable instead of hardcodingPAGE_SIZEindependently inceph_osd_data_pages_init()and themin_t()cap, per Viacheslav Dubeyko's review.
---
net/ceph/osd_client.c | 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)net/ceph/osd_client.c +1
-0@@ -5063,6 +5063,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, struct ceph_osd_request *req; struct page **pages; int ret; +const size_t buf_len =PAGE_SIZE ;
req =ceph_osdc_alloc_request(osdc, NULL, 1, false, GFP_NOIO);
if (!req)net/ceph/osd_client.c +1
-1@@ -5081,7 +5082,7 @@ int ceph_osdc_list_watchers(structceph_osd_client *osdc, osd_req_op_init(req, 0, CEPH_OSD_OP_LIST_WATCHERS, 0); ceph_osd_data_pages_init(osd_req_op_data(req, 0, list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true);
ret =ceph_osdc_alloc_messages(req, GFP_NOIO); if (ret)
net/ceph/osd_client.c int ceph_osdc_list_watchers(struct · line 5099
Quoted 9 lines
net/ceph/osd_client.c +2
-1@@ -5091,7 +5092,8 @@ int ceph_osdc_list_watchers(structceph_osd_client *osdc, ret = ceph_osdc_wait_request(osdc, req); if (ret >= 0) { void *p = page_address(pages[0]); -void *const end = p +min_t (u32, req- >r_ops[0].outdata_len, PAGE_SIZE); +void *const end = p + +min_t (u32, req->r_ops[0 ].outdata_len, buf_len);
Now, min_t() worries me slightly because req->r_ops[0].outdata_len is u32 data type, but buf_len is size_t. Could we have the same data type for both variables?
Thanks,
Slava.
Quoted 4 lines
ret =decode_watchers(&p, end, watchers, num_watchers);
}
PATCH v3 1/1 ceph: fix OOB read in ceph_osdc_list_watchers via uncapped outdata_len
The OSD reply header field op->payload_len is wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:
m->outdata_len[i] = le32_to_cpu(op->payload_len);
This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in ceph_osdc_list_watchers():
void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO) ceph_osd_data_pages_init(..., PAGE_SIZE, ...)
The messenger caps the copy to PAGE_SIZE bytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing _safe decoder boundary checks to pass while the physical reads cross the slab allocation boundary.
KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
Read of size 4 at addr ffff88800a229f9e by task insmod/57
CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy)
Tainted: [O]=OOT_MODULE
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4d/0x70
print_report+0x170/0x4f3
? __pfx__raw_spin_lock_irqsave+0x10/0x10
kasan_report+0xda/0x110
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc]
ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
? __pfx_do_one_initcall+0x10/0x10
? kasan_unpoison+0x44/0x70
do_init_module+0x27c/0x790
? __pfx_do_init_module+0x10/0x10
? __kasan_slab_free+0x47/0x70
? kfree+0x15f/0x3b0
load_module+0x4a9a/0x6350
? __pfx_load_module+0x10/0x10
? security_file_permission+0x24/0x50
? kernel_read_file+0x2ed/0x770
? init_module_from_file+0x15c/0x180
init_module_from_file+0x15c/0x180
? __pfx_init_module_from_file+0x10/0x10
? tick_nohz_handler+0x2a3/0x640
? _raw_spin_lock+0x7e/0xd0
idempotent_init_module+0x21f/0x750
? __pfx_idempotent_init_module+0x10/0x10
? fdget+0x4e/0x4a0
? fdget+0x4e/0x4a0
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
? exc_page_fault+0x66/0xb0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Allocated by task 57:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_kmalloc+0x7f/0x90
ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
do_init_module+0x27c/0x790
load_module+0x4a9a/0x6350
init_module_from_file+0x15c/0x180
idempotent_init_module+0x21f/0x750
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88800a229000
which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 3998 bytes inside of
allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0)
Memory state around the buggy address:
ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing buf_len to hold the allocation size, using it in both ceph_osd_data_pages_init() and the min_t() decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes. buf_len is declared as u32 to match the type of outdata_len used in the min_t() expression.
Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing CEPH_OSD_OP_LIST_WATCHERS without further privileges beyond OSD session establishment.
---
v3: Change buf_len type from size_t to u32 to match outdata_len type
in min_t(), per Viacheslav Dubeyko's review.
v2: Introduce buf_len variable instead of hardcoding PAGE_SIZE
independently in ceph_osd_data_pages_init() and the min_t() cap, per Viacheslav Dubeyko's review.
--- net/ceph/osd_client.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +4 -2
@@ -5063,6 +5063,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_osd_request *req;struct page **pages;int ret; +const u32 buf_len =PAGE_SIZE ; req =ceph_osdc_alloc_request (osdc,NULL ,1 , false,GFP_NOIO );if (!req)
@@ -5081,7 +5082,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,osd_req_op_init (req,0 ,CEPH_OSD_OP_LIST_WATCHERS ,0 );ceph_osd_data_pages_init (osd_req_op_data (req,0 , list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true); ret =ceph_osdc_alloc_messages (req,GFP_NOIO );if (ret)
@@ -5091,7 +5092,8 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p +min_t (u32, req->r_ops[0 ].outdata_len,PAGE_SIZE ); +void *const end = p + +min_t (u32, req->r_ops[0 ].outdata_len, buf_len); ret =decode_watchers (&p, end, watchers, num_watchers); }
Signature · 1 line
2.53.0
Quoted 156 lines · Pavitra Jha
The OSD reply header field
op->payload_lenis wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:m->outdata_len[i] =
le32_to_cpu(op->payload_len);This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in
ceph_osdc_list_watchers():void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO)ceph_osd_data_pages_init(..., PAGE_SIZE, ...)The messenger caps the copy to
PAGE_SIZEbytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing_safedecoder boundary checks to pass while the physical reads cross the slab allocation boundary.KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] Read of size 4 at addr ffff88800a229f9e by task insmod/57 CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy) Tainted: [O]=OOT_MODULE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0- debian-1.17.0-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4d/0x70 print_report+0x170/0x4f3 ? __pfx__raw_spin_lock_irqsave+0x10/0x10 kasan_report+0xda/0x110 ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] ? __pfx_ceph_oob2_init+0x10/0x10 [ceph_oob2_poc] ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 ? __pfx_do_one_initcall+0x10/0x10 ? kasan_unpoison+0x44/0x70 do_init_module+0x27c/0x790 ? __pfx_do_init_module+0x10/0x10 ? __kasan_slab_free+0x47/0x70 ? kfree+0x15f/0x3b0 load_module+0x4a9a/0x6350 ? __pfx_load_module+0x10/0x10 ? security_file_permission+0x24/0x50 ? kernel_read_file+0x2ed/0x770 ? init_module_from_file+0x15c/0x180 init_module_from_file+0x15c/0x180 ? __pfx_init_module_from_file+0x10/0x10 ? tick_nohz_handler+0x2a3/0x640 ? _raw_spin_lock+0x7e/0xd0 idempotent_init_module+0x21f/0x750 ? __pfx_idempotent_init_module+0x10/0x10 ? fdget+0x4e/0x4a0 ? fdget+0x4e/0x4a0 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 ? exc_page_fault+0x66/0xb0 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 57: kasan_save_stack+0x30/0x50 kasan_save_track+0x14/0x30 __kasan_kmalloc+0x7f/0x90 ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 do_init_module+0x27c/0x790 load_module+0x4a9a/0x6350 init_module_from_file+0x15c/0x180 idempotent_init_module+0x21f/0x750 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 entry_SYSCALL_64_after_hwframe+0x77/0x7f The buggy address belongs to the object at ffff88800a229000which belongs to the cache kmalloc-4k of size 4096 The buggy address is located 3998 bytes inside of allocated 4000-byte region [
ffff88800a229000,ffff88800a229fa0)Memory state around the buggy address: ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing
buf_lento hold the allocation size, using it in bothceph_osd_data_pages_init()and themin_t()decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes.buf_lenis declared as u32 to match the type ofoutdata_lenused in themin_t()expression.Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing
CEPH_OSD_OP_LIST_WATCHERSwithout further privileges beyond OSD session establishment.Fixes a4ed38d7a180 ("libceph: support for
CEPH_OSD_OP_LIST_WATCHERS")Signed-off-by Pavitra Jha <[email protected]>---
v3: Changebuf_lentype fromsize_tto u32 to matchoutdata_lentype inmin_t(), per Viacheslav Dubeyko's review.
v2: Introducebuf_lenvariable instead of hardcodingPAGE_SIZEindependently inceph_osd_data_pages_init()and themin_t()cap, per Viacheslav Dubeyko's review.
---
net/ceph/osd_client.c | 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)net/ceph/osd_client.c +1
-0@@ -5063,6 +5063,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, struct ceph_osd_request *req; struct page **pages; int ret; +const u32 buf_len =PAGE_SIZE ;
req =ceph_osdc_alloc_request(osdc, NULL, 1, false, GFP_NOIO);
if (!req)net/ceph/osd_client.c +1
-1@@ -5081,7 +5082,7 @@ int ceph_osdc_list_watchers(structceph_osd_client *osdc, osd_req_op_init(req, 0, CEPH_OSD_OP_LIST_WATCHERS, 0); ceph_osd_data_pages_init(osd_req_op_data(req, 0, list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true);
ret =ceph_osdc_alloc_messages(req, GFP_NOIO); if (ret)net/ceph/osd_client.c +2
-1@@ -5091,7 +5092,8 @@ int ceph_osdc_list_watchers(structceph_osd_client *osdc, ret = ceph_osdc_wait_request(osdc, req); if (ret >= 0) { void *p = page_address(pages[0]); -void *const end = p +min_t (u32, req- >r_ops[0].outdata_len, PAGE_SIZE); +void *const end = p + +min_t (u32, req->r_ops[0 ].outdata_len, buf_len);
Quoted 4 lines
ret =decode_watchers(&p, end, watchers, num_watchers);
}
Looks good.
Thanks,
Slava.
Alex Markuze
maintainerHi Pavitra,
NACK for now: this patch does not apply cleanly to the current testing branch or linux/master. Please rebase and resend a new version.
I tested it with git am -3 against both trees, and both attempts failed.
Signature · 1 line
Alex Markuze
PATCH v4 1/1 ceph: fix OOB read in ceph_osdc_list_watchers via uncapped outdata_len
The OSD reply header field op->payload_len is wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:
m->outdata_len[i] = le32_to_cpu(op->payload_len);
This value propagates unchecked to req->r_ops[0].outdata_len and is then used to set the decode boundary in ceph_osdc_list_watchers():
void *const end = p + req->r_ops[0].outdata_len;
The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO) ceph_osd_data_pages_init(..., PAGE_SIZE, ...)
The messenger caps the copy to PAGE_SIZE bytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send outdata_len=0x10000, causing _safe decoder boundary checks to pass while the physical reads cross the slab allocation boundary. KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
Read of size 4 at addr ffff88800a229f9e by task insmod/57
CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy)
Tainted: [O]=OOT_MODULE
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4d/0x70
print_report+0x170/0x4f3
kasan_report+0xda/0x110
ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
do_init_module+0x27c/0x790
load_module+0x4a9a/0x6350
init_module_from_file+0x15c/0x180
idempotent_init_module+0x21f/0x750
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Allocated by task 57:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_kmalloc+0x7f/0x90
ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc]
do_one_initcall+0x9a/0x3a0
do_init_module+0x27c/0x790
load_module+0x4a9a/0x6350
init_module_from_file+0x15c/0x180
idempotent_init_module+0x21f/0x750
__x64_sys_finit_module+0xba/0x120
do_syscall_64+0xe2/0x570
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88800a229000
which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 3998 bytes inside of
allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0)
Memory state around the buggy address:
ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing buf_len to hold the allocation size, using it in both ceph_osd_data_pages_init() and the min_t() decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes. buf_len is declared as u32 to match the type of outdata_len used in the min_t() expression.
Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing CEPH_OSD_OP_LIST_WATCHERS without further privileges beyond OSD session establishment.
---
v4: Rebase against current linux/master and ceph-testing/testing,
confirmed clean git am -3 apply against both. No functional changes from Slava's reviewed v3.
v3: Change buf_len type from size_t to u32 to match outdata_len type
in min_t(), per Viacheslav Dubeyko's review.
v2: Introduce buf_len variable instead of hardcoding PAGE_SIZE
independently in ceph_osd_data_pages_init() and the min_t() cap, per Viacheslav Dubeyko's review.
--- net/ceph/osd_client.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +4 -2
@@ -5060,6 +5060,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_osd_request *req;struct page **pages;int ret; +const u32 buf_len =PAGE_SIZE ; req =ceph_osdc_alloc_request (osdc,NULL ,1 , false,GFP_NOIO );if (!req)
@@ -5078,7 +5079,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,osd_req_op_init (req,0 ,CEPH_OSD_OP_LIST_WATCHERS ,0 );ceph_osd_data_pages_init (osd_req_op_data (req,0 , list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true); ret =ceph_osdc_alloc_messages (req,GFP_NOIO );if (ret)
@@ -5088,7 +5089,8 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p + req->r_ops[0 ].outdata_len; +void *const end = p + +min_t (u32, req->r_ops[0 ].outdata_len, buf_len); ret =decode_watchers (&p, end, watchers, num_watchers); }
Signature · 1 line
2.53.0
Ilya Dryomov
maintainerQuoted 7 lines · Pavitra Jha
The OSD reply header field
op->payload_lenis wire-controlled and is copied directly into m->outdata_len[i] without any bounds check:m->outdata_len[i] = le32_to_cpu(op->payload_len);This value propagates unchecked to req->r_ops[0].outdata_len and is
Hi Pavitra,
IIRC it's not completely unchecked: if the sum of m.outdata_len values for all OSD ops in the request doesn't match msg->hdr.data_len, handle_reply() should fail and hence ceph_osdc_wait_request() in ceph_osdc_list_watchers() too.
Quoted 10 lines
then used to set the decode boundary in
ceph_osdc_list_watchers():void *const end = p + req->r_ops[0].outdata_len;The actual data allocation is always exactly one page:
ceph_alloc_page_vector(1, GFP_NOIO)ceph_osd_data_pages_init(..., PAGE_SIZE, ...)The messenger caps the copy to
PAGE_SIZEbytes, but the decode window end is set from the uncapped wire value. A malicious OSD can send
I don't think the messenger ever silently caps anything (or at least it shouldn't). The messenger wants to read exactly msg->hdr.data_len bytes off the wire and if the underlying buffers aren't big enough the message gets discarded.
Quoted 3 lines
outdata_len=0x10000, causing
_safedecoder boundary checks to pass while the physical reads cross the slab allocation boundary. KASAN report (kernel 7.0.0-rc7, QEMU/x86_64, KASLR disabled):
Could you please elaborate on what fields were set to what values and what payload was injected and fed to decode_watchers() because it for sure wasn't just outdata_len set to 0x10000.
Thanks,
Ilya
Quoted 115 lines
==================================================================
BUG: KASAN: slab-out-of-bounds in ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] Read of size 4 at addr ffff88800a229f9e by task insmod/57 CPU: 0 UID: 0 PID: 57 Comm: insmod Tainted: G O 7.0.0-rc7-g9c2abf69da83-dirty #15 PREEMPT(lazy) Tainted: [O]=OOT_MODULE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4d/0x70 print_report+0x170/0x4f3 kasan_report+0xda/0x110 ceph_oob2_init+0x23d/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 do_init_module+0x27c/0x790 load_module+0x4a9a/0x6350 init_module_from_file+0x15c/0x180 idempotent_init_module+0x21f/0x750 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 57: kasan_save_stack+0x30/0x50 kasan_save_track+0x14/0x30 __kasan_kmalloc+0x7f/0x90 ceph_oob2_init+0x44/0xff0 [ceph_oob2_poc] do_one_initcall+0x9a/0x3a0 do_init_module+0x27c/0x790 load_module+0x4a9a/0x6350 init_module_from_file+0x15c/0x180 idempotent_init_module+0x21f/0x750 __x64_sys_finit_module+0xba/0x120 do_syscall_64+0xe2/0x570 entry_SYSCALL_64_after_hwframe+0x77/0x7f The buggy address belongs to the object at ffff88800a229000 which belongs to the cache kmalloc-4k of size 4096 The buggy address is located 3998 bytes inside of allocated 4000-byte region [ffff88800a229000, ffff88800a229fa0) Memory state around the buggy address: ffff88800a229e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffff88800a229f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >ffff88800a229f80: 00 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc ^ ffff88800a22a000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff88800a22a080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ==================================================================val=0xccccaaaa (OOB garbage from KASAN redzone)
Fix by introducing
buf_lento hold the allocation size, using it in bothceph_osd_data_pages_init()and themin_t()decode boundary cap, so the two are guaranteed to stay in sync if the buffer size changes.buf_lenis declared as u32 to match the type ofoutdata_lenused in themin_t()expression.Attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any client issuing
CEPH_OSD_OP_LIST_WATCHERSwithout further privileges beyond OSD session establishment.Fixes a4ed38d7a180 ("libceph: support for CEPH_OSD_OP_LIST_WATCHERS")Reviewed-by Viacheslav Dubeyko <[email protected]>Signed-off-by Pavitra Jha <[email protected]>---
v4: Rebase against current linux/master and ceph-testing/testing,
confirmed clean git am -3 apply against both. No functional changes from Slava's reviewed v3.
v3: Changebuf_lentype fromsize_tto u32 to matchoutdata_lentype
inmin_t(), per Viacheslav Dubeyko's review.
v2: Introducebuf_lenvariable instead of hardcodingPAGE_SIZE
independently inceph_osd_data_pages_init()and themin_t()cap, per Viacheslav Dubeyko's review.--- net/ceph/osd_client.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)net/ceph/osd_client.c +4
-3@@ -5060,6 +5060,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_osd_request *req;struct page **pages;int ret; +const u32 buf_len =PAGE_SIZE ; req =ceph_osdc_alloc_request (osdc,NULL ,1 , false,GFP_NOIO );if (!req)@@ -5078,7 +5079,7 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,osd_req_op_init (req,0 ,CEPH_OSD_OP_LIST_WATCHERS ,0 );ceph_osd_data_pages_init (osd_req_op_data (req,0 , list_watchers, response_data), - pages,PAGE_SIZE ,0 , false, true); + pages, buf_len,0 , false, true); ret =ceph_osdc_alloc_messages (req,GFP_NOIO );if (ret)@@ -5088,7 +5089,8 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, ret =ceph_osdc_wait_request (osdc, req);if (ret >=0 ) {void *p =page_address (pages[0 ]); -void *const end = p + req->r_ops[0 ].outdata_len; +void *const end = p + +min_t (u32, req->r_ops[0 ].outdata_len, buf_len); ret =decode_watchers (&p, end, watchers, num_watchers); } --2.53.0