Selinux/hooks.c: Fix a NULL pointer dereference caused by semop()
PATCH 1/1 Selinux/hooks.c: Fix a NULL pointer dereference caused by semop()
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206
Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d
45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000
The root cause is semtimedop() was called after semget() without checking its return value in process opcmon. and semget() failed to check permission in function avc_has_perm() then sem_perm->security was freed shown as following:
sys_semget()
->newary()
->security_sem_alloc()
->sem_alloc_security()
selinux_sem_alloc_security()
->ipc_alloc_security() {
->rc = avc_has_perm()
if (rc) {
ipc_free_security(&sma->sem_perm);
return rc;
So ipc_perms->security was NULL, then semtimedop() was called as
following:
sys_semtimedop() / semop()
->selinux_sem_semop()
->ipc_has_perm()
->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);^- NULL pointer dereference happens
The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.
--- security/selinux/hooks.c | 2 ++ 1 file changed, 2 insertions(+)
security/selinux/hooks.c +2 -0
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,u32 sid =current_sid (); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ; ad.u.ipc_id = ipc_perms->key;
Signature · 1 line
1.8.3.1
Quoted 43 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<
ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its return value in process opcmon. andsemget()failed to check permission in functionavc_has_perm()thensem_perm->securitywas freed shown as following:
Quoted 10 lines
sys_semget()
->newary()
->security_sem_alloc()
->sem_alloc_security()
selinux_sem_alloc_security()
->ipc_alloc_security() {
->rc = avc_has_perm()
if (rc) {
ipc_free_security(&sma->sem_perm);
return rc;We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?
Quoted 24 lines
So ipc_perms->security was NULL, then semtimedop() was called as following: sys_semtimedop() / semop() ->selinux_sem_semop() ->ipc_has_perm() ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);^- NULL pointer dereference happens
The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.Signed-off-by Ethan Zhao <[email protected]>--- security/selinux/hooks.c | 2 ++ 1 file changed, 2 insertions(+)security/selinux/hooks.c +0
-0
security/selinux/hooks.c static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, · line 5136
Quoted 8 lines
security/selinux/hooks.c +2
-0@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,u32 sid =current_sid (); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ; ad.u.ipc_id = ipc_perms->key;
That is not the correct fix; it just hides a bug. If we reach ipc_has_perm() with a NULL isec, it is a bug in the ipc code.
Hi,
Quoted 54 lines · Stephen Smalley
Quoted 43 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<
ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its return value in process opcmon. andsemget()failed to check permission in functionavc_has_perm()thensem_perm->securitywas freed shown as following:Quoted 10 lines
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?
My only idea would be a race of semtimedop() with IPC_RMID:
If a rcu grace period happens between sem_obtain_object_check() and the ipc_has_perm() call, the the observed NULL pointer assignment would happen.
Quoted 10 lines
Quoted 10 lines
So ipc_perms->security was NULL, then semtimedop() was called as following: sys_semtimedop() / semop() ->selinux_sem_semop() ->ipc_has_perm() ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);^- NULL pointer dereference happens
The test kernel was running on VMware.
Can you give some more details?
- which RCU type?
- PREEMPT?
--
Manfred
Quoted 67 lines · Manfred Spraul
Hi,
Quoted 58 lines · Stephen Smalley
Quoted 47 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8
<0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its
return value in process opcmon. andsemget()failed to check permission in
functionavc_has_perm()thensem_perm->securitywas freed shown as following:Quoted 10 lines
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?
Quoted 3 lines
My only idea would be a race of
semtimedop()withIPC_RMID:
If a rcu grace period happens betweensem_obtain_object_check()and theipc_has_perm()call, the the observed NULL pointer assignment would happen.
We only free and clear the ipc_perms->security field on a failure during newary() -> security_sem_alloc(), in which case we fail with an error before the ipc_addid() call has occurred, or during sem_rcu_free() ->security_sem_free() just prior to calling ipc_rcu_free(). So I don't
see how ipc_perms->security can be NULL in ipc_has_perm(). We could rcu free the ipc_perms->security field but I don't see why that would be correct/necessary.
What kernel version was this? Didn't we have this problem and solve it upstream some time ago? IPC could be allocated with a valid security context, the ipc would be freed. the isec was free'd syncronously, but then the ipc could stick around until some rcu period or some usage flag got to 0, then it got freed...
this seems so familiar, but it was a while ago
Quoted 80 lines · Stephen Smalley
Quoted 67 lines · Manfred Spraul
Hi,
Quoted 58 lines · Stephen Smalley
Quoted 47 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8
<0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its
return value in process opcmon. andsemget()failed to check permission in
functionavc_has_perm()thensem_perm->securitywas freed shown as following:Quoted 10 lines
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Quoted 3 lines
My only idea would be a race of
semtimedop()withIPC_RMID:
If a rcu grace period happens betweensem_obtain_object_check()and theipc_has_perm()call, the the observed NULL pointer assignment would happen.We only free and clear the
ipc_perms->securityfield on a failure duringnewary()->security_sem_alloc(), in which case we fail with an error before theipc_addid()call has occurred, or duringsem_rcu_free()->security_sem_free()just prior to callingipc_rcu_free(). So I don't
see howipc_perms->securitycan be NULL inipc_has_perm(). We could rcu free theipc_perms->securityfield but I don't see why that would be correct/necessary.
Quoted 7 lines · Eric Paris
What kernel version was this? Didn't we have this problem and solve it upstream some time ago? IPC could be allocated with a valid security context, the ipc would be freed. the isec was free'd syncronously, but then the ipc could stick around until some rcu period or some usage flag got to 0, then it got freed...
this seems so familiar, but it was a while ago
Don't know the kernel version.
Are you thinking of the inode security bug? Because in that case the vfs changed underneath us to rcu free the inode but left the security_inode_free() call outside of the rcu callback (due to the rcu callback only being used in the default case where the filesystem does not implement its own destroy_inode() method). In this case, we don't seem to have that situation AFAICS.
Quoted 85 lines · Stephen Smalley
Quoted 80 lines · Manfred Spraul
Quoted 67 lines
Hi,
Quoted 58 lines · Stephen Smalley
Quoted 47 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8
<0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its
return value in process opcmon. andsemget()failed to check permission in
functionavc_has_perm()thensem_perm->securitywas freed shown as following:Quoted 10 lines
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Quoted 3 lines
My only idea would be a race of
semtimedop()withIPC_RMID:
If a rcu grace period happens betweensem_obtain_object_check()and theipc_has_perm()call, the the observed NULL pointer assignment would happen.We only free and clear the
ipc_perms->securityfield on a failure duringnewary()->security_sem_alloc(), in which case we fail with an error before theipc_addid()call has occurred, or duringsem_rcu_free()->security_sem_free()just prior to callingipc_rcu_free(). So I don't
see howipc_perms->securitycan be NULL inipc_has_perm(). We could rcu free theipc_perms->securityfield but I don't see why that would be correct/necessary.
Stephen,
Quoted 76 lines · Manfred Spraul
Hi,
Quoted 54 lines · Stephen Smalley
Quoted 43 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<
ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its return value in process opcmon. andsemget()failed to check permission in functionavc_has_perm()thensem_perm->securitywas freed shown as following:Quoted 10 lines
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?My only idea would be a race of
semtimedop()withIPC_RMID:
If a rcu grace period happens betweensem_obtain_object_check()and theipc_has_perm()call, the the observed NULL pointer assignment would happen.Quoted 10 lines
Quoted 10 lines
So ipc_perms->security was NULL, then semtimedop() was called as following: sys_semtimedop() / semop() ->selinux_sem_semop() ->ipc_has_perm() ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad); ^- NULL pointerdereference happens
Quoted 4 lines
Quoted 1 line
Quoted 1 line
The test kernel was running on VMware.
Can you give some more details?
- which RCU type?
- PREEMPT?
Let me figure out more detail about it and share with you.
Thanks,
Ethan
Quoted 3 lines
Signature · 2 lines
-- Manfred
Quoted 55 lines · Stephen Smalley
Quoted 43 lines · Ethan Zhao
A NULL pointer dereference was observed as following panic:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60 ... Process opcmon (pid: 30712, threadinfo ffff880237f2a000, task ffff88022ac70e40) Stack:
ffff880237f2bc04ffffffff01020953ffff880237f2bce8ffffffff8125818e
00000000000000010000000037f78004ffff880237f2bcd8ffffffff81273619ffff880237f2bce8ffffffff8126e3e6ffff880237f2bf68ffffffff8125c206Call Trace: [<ffffffff8125818e>] ? ipcperms+0xae/0x110 [<ffffffff81273619>] selinux_sem_semop+0x19/0x20 [<ffffffff8126e3e6>] security_sem_semop+0x16/0x20 [<ffffffff8125c206>] sys_semtimedop+0x346/0x750 [<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff8109a763>] ? up_read+0x23/0x40 [<ffffffff8161d830>] ? __do_page_fault+0x280/0x500 [<ffffffff81182f1c>] ? might_fault+0x5c/0xb0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0 [<ffffffff81081f96>] ? sys_newuname+0x66/0xf0 [<ffffffff81622f45>] ? sysret_check+0x22/0x5d [<ffffffff8125c620>] sys_semop+0x10/0x20 [<ffffffff81622f19>] system_call_fastpath+0x16/0x1b Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<
ffffffff812735eb>]ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000The root cause is
semtimedop()was called aftersemget()without checking its return value in process opcmon. andsemget()failed to check permission in functionavc_has_perm()thensem_perm->securitywas freed shown as following:Quoted 10 lines
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?
Seems it wouldn't happen after commit e8577d1f0329d4842e8302e289fb2c22156abef4 ?
Thanks,
Ethan
Quoted 42 lines
Quoted 25 lines
So ipc_perms->security was NULL, then semtimedop() was called as following: sys_semtimedop() / semop() ->selinux_sem_semop() ->ipc_has_perm() ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad); ^- NULL pointer dereference happensThe test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.Signed-off-by Ethan Zhao <[email protected]>--- security/selinux/hooks.c | 2 ++ 1 file changed, 2 insertions(+)security/selinux/hooks.c +0
-0security/selinux/hooks.c static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, · line 5136
Quoted 8 lines
security/selinux/hooks.c +2
-0@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,u32 sid =current_sid (); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ; ad.u.ipc_id = ipc_perms->key;That is not the correct fix; it just hides a bug. If we reach
ipc_has_perm()with a NULL isec, it is a bug in the ipc code.--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 12 lines · Ethan Zhao
Quoted 11 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) { ipc_free_security(&sma->sem_perm); return rc;
Quoted 7 lines
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?
That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.
thread A:
thread B:
semtimedop()
-> sem_obtain_object_check()
semctl(IPC_RMID)
-> freeary()
-> ipc_rcu_putref()
-> call_rcu()
-> somehow a grace period
-> sem_rcu_free()
-> security_sem_free()Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?
--
Manfred
Quoted 42 lines
Thanks,
EthanQuoted 40 lines
Quoted 24 lines
So ipc_perms->security was NULL, then semtimedop() was called as following: sys_semtimedop() / semop() ->selinux_sem_semop() ->ipc_has_perm() ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad); ^- NULL pointer dereference happensThe test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.Signed-off-by Ethan Zhao <[email protected]>--- security/selinux/hooks.c | 2 ++ 1 file changed, 2 insertions(+)security/selinux/hooks.c +0
-0security/selinux/hooks.c static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, · line 5136
Quoted 8 lines
security/selinux/hooks.c +2
-0@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,u32 sid =current_sid (); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ; ad.u.ipc_id = ipc_perms->key;That is not the correct fix; it just hides a bug. If we reach
ipc_has_perm()with a NULL isec, it is a bug in the ipc code.--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 46 lines · Manfred Spraul
Quoted 25 lines · Ethan Zhao
Quoted 15 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {Quoted 2 lines
ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Quoted 2 lines
Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()
Quoted 2 lines
Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?
I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
Thanks,
Ethan
Quoted 54 lines
--
ManfredQuoted 47 lines
Thanks,
EthanQuoted 45 lines
Quoted 34 lines
So ipc_perms->security was NULL, then semtimedop() was called as following: sys_semtimedop() / semop() ->selinux_sem_semop() ->ipc_has_perm() ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad); ^- NULL pointer dereferencehappens
The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space.
This patch was tested with v3.19-rc5.Signed-off-by Ethan Zhao <[email protected]>--- security/selinux/hooks.c | 2 ++ 1 file changed, 2 insertions(+)security/selinux/hooks.c +2
-0@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, u32 sid = current_sid(); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ;Quoted 2 lines
ad.u.ipc_id = ipc_perms->key;That is not the correct fix; it just hides a bug. If we reach
ipc_has_perm()with a NULL isec, it is a bug in the ipc code.--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {ipc_free_security(&sma->sem_perm); return rc;Quoted 8 lines
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()
Quoted 4 lines
Quoted 2 lines
Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
Is the issue reproducable?
If yes, can you try something like the attached patch?
Quoted 11 lines
security/selinux/hooks.c +2
-0@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, u32 sid = current_sid(); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ;
Quoted 1 line
ad.u.ipc_id = ipc_perms->key;
ipc_has_perm() runs without any spinlocks/semaphores, only rcu_read_lock().
Testing for ipc_perms->security!=NULL does solve the issue that ipc_perm->key could be an access to kfree'd memory: Nothing prevents that the kfree could happen just after the test.
I.e.: The patch can't be the right solution.
--
Manfred
ipc/sem.c +6 -0
@@ -248,6 +248,7 @@ static void sem_rcu_free(struct rcu_head *head) struct ipc_rcu *p = container_of(head, struct ipc_rcu, rcu);struct sem_array *sma =ipc_rcu_to_struct (p); +pr_info ("sem_rcu_free: sma %p\n" ,sma);security_sem_free (sma);ipc_rcu_free (head); }
@@ -529,6 +530,7 @@ static int newary(struct ipc_namespace *ns, struct ipc_params *params) sma->sem_nsems = nsems; sma->sem_ctime =get_seconds (); +pr_info ("newary: sma %p becomes visible\n" ,sma); id =ipc_addid (&sem_ids (ns), &sma->sem_perm, ns->sc_semmni);if (id <0 ) {ipc_rcu_putref (sma, sem_rcu_free);
@@ -1118,6 +1120,7 @@ static void freeary(struct ipc_namespace *ns, struct kern_ipc_perm *ipcp)/* Remove the semaphore set from the IDR */ sem_rmid (ns, sma); +pr_info ("freeary: sma %p unlinked\n" ,sma);sem_unlock (sma,-1 );rcu_read_unlock ();
@@ -1860,6 +1863,9 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,if (ipcperms (ns, &sma->sem_perm, alter ?S_IWUGO :S_IRUGO )) goto out_rcu_wakeup; +if (sma->sem_perm.security ==NULL ) { +pr_info ("sma %p: sem_perm.security == NULL\n" , sma); + } error =security_sem_semop (sma, sops, nsops, alter);if (error) goto out_rcu_wakeup;
security/selinux/hooks.c +8 -0
@@ -5088,6 +5088,7 @@ static int ipc_alloc_security(struct task_struct *task, isec->sclass = sclass; isec->sid = sid; perm->security = isec; +pr_info ("ipc_alloc_security for perm %p.\n" , perm);return 0 ; }
@@ -5096,6 +5097,7 @@ static void ipc_free_security(struct kern_ipc_perm *perm) {struct ipc_security_struct *isec = perm->security; perm->security =NULL ; +pr_info ("ipc_free_security for perm %p.\n" , perm);kfree (isec); }
@@ -5129,6 +5131,12 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,u32 sid =current_sid (); isec = ipc_perms->security; +if (isec ==NULL ) { + struct sem_array *sma = container_of(ipc_perms, struct sem_array, sem_perm); + +pr_info ("sma %p, sem_base %p, deleted %d with NULL isec\n" , + sma, sma->sem_base, sma->sem_perm.deleted); + } ad.type =LSM_AUDIT_DATA_IPC ; ad.u.ipc_id = ipc_perms->key;
Manfred,
Quoted 43 lines · Manfred Spraul
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {ipc_free_security(&sma->sem_perm); return rc;Quoted 8 lines
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
Is the issue reproducable?
It was hit on an user's VMware while running a process named "opcmon"
maybe ported from HP_UX. I asked for the vmwcore, but not get it yet.
Quoted 1 line
If yes, can you try something like the attached patch?
Yes, will.
Thanks,
Ethan
Quoted 20 lines
Quoted 12 lines
security/selinux/hooks.c +2
-0@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, u32 sid = current_sid(); isec = ipc_perms->security; +if (!isec) +return -EACCES ; ad.type =LSM_AUDIT_DATA_IPC ;Quoted 1 line
ad.u.ipc_id = ipc_perms->key;
ipc_has_perm()runs without any spinlocks/semaphores, onlyrcu_read_lock().
Testing for ipc_perms->security!=NULL does solve the issue thatipc_perm->keycould be an access to kfree'd memory: Nothing prevents that the kfree could happen just after the test.I.e.: The patch can't be the right solution.
Signature · 1 line
Manfred
Quoted 48 lines · Ethan Zhao
Quoted 46 lines · Manfred Spraul
Quoted 25 lines · Ethan Zhao
Quoted 15 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {Quoted 2 lines
ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Quoted 2 lines
Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()
Quoted 5 lines
Quoted 2 lines
Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
You said the patch was tested with 3.19-rc5. But did you reproduce the bug on that kernel version before the patch? If not, what kernel version were you running when you triggered the bug?
Quoted 52 lines · Stephen Smalley
Quoted 51 lines · Ethan Zhao
Quoted 48 lines · Manfred Spraul
Quoted 25 lines · Ethan Zhao
Quoted 15 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {Quoted 2 lines
ipc_free_security(&sma->sem_perm); return rc;Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Quoted 2 lines
Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?
Quoted 6 lines
Quoted 2 lines
I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
You said the patch was tested with 3.19-rc5. But did you reproduce the bug on that kernel version before the patch? If not, what kernel version were you running when you triggered the bug?
Also, is this a vanilla kernel? Or from a distro?
Essentially, did the kernel with the reproducible bug have:
commit 53dad6d3a8e5ac1af8bacc6ac2134ae1a8b085f1
Author: Davidlohr Bueso <[email protected]> Date: Mon Sep 23 17:04:45 2013 -0700
ipc: fix race with LSMs
Currently, IPC mechanisms do security and auditing related checks under RCU. However, since security modules can free the security structure, for example, through selinux_[sem,msg_queue,shm]_free_security(), we can race if the structure is freed before other tasks are done with it, creating a use-after-free condition. Manfred illustrates this nicely, for instance with shared mem and selinux:
-> do_shmat calls rcu_read_lock()
-> do_shmat calls shm_object_check().
Checks that the object is still valid - but doesn't acquire any locks. Then it returns.
-> do_shmat calls security_shm_shmat (e.g. selinux_shm_shmat) -> selinux_shm_shmat calls ipc_has_perm()
-> ipc_has_perm accesses ipc_perms->security
shm_close()
-> shm_close acquires rw_mutex & shm_lock
-> shm_close calls shm_destroy
-> shm_destroy calls security_shm_free (e.g. selinux_shm_free_security) -> selinux_shm_free_security calls ipc_free_security(&shp->shm_perm) -> ipc_free_security calls kfree(ipc_perms->security)
This patch delays the freeing of the security structures after all RCU readers are done. Furthermore it aligns the security life cycle with that of the rest of IPC - freeing them based on the reference counter. For situations where we need not free security, the current behavior is kept. Linus states:
"... the old behavior was suspect for another reason too: having the
security blob go away from under a user sounds like it could cause various other problems anyway, so I think the old code was at least _prone_ to bugs even if it didn't have catastrophic behavior."
I have tested this patch with IPC testcases from LTP on both my quad-core laptop and on a 64 core NUMA server. In both cases selinux is enabled, and tests pass for both voluntary and forced preemption models. While the mentioned races are theoretical (at least no one as reported them), I wanted to make sure that this new logic doesn't break anything we weren't aware of.
Additionally, Manfred's concerns about the grace period are quite valid, and it obviously assumes that the ->security nil dereference issue still exists to some extent. Changes in RCU are something to consider as well. This is all pretty iffy though, lets make sure we are looking at the right kernel first.
Thanks,
Davidlohr
Davidlohr,
Quoted 44 lines · Davidlohr Bueso
Quoted 43 lines · Stephen Smalley
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {ipc_free_security(&sma->sem_perm); return rc;Quoted 8 lines
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
You said the patch was tested with 3.19-rc5. But did you reproduce the bug on that kernel version before the patch? If not, what kernel version were you running when you triggered the bug?
Also, is this a vanilla kernel? Or from a distro?
The hard thing, it is hit on customer's environment, the issue kernel
doesn't
have many commits far from the last about IPC/SElinux.
Quoted 1 line
Essentially, did the kernel with the reproducible bug have:
Not easy to do reproduce it is triggered by a process "opcmon" not
public to everyone.
What I have is the panic log. the vmware not get yet.
Thanks,
Ethan
Quoted 5 lines
commit
53dad6d3a8e5ac1af8bacc6ac2134ae1a8b085f1Author: Davidlohr Bueso <[email protected]> Date: Mon Sep 23 17:04:45 2013 -0700ipc: fix race with LSMs
No, the kernel doesn't have this commit, will try it.
Quoted 42 lines
Currently, IPC mechanisms do security and auditing related checks under RCU. However, since security modules can free the security structure, for example, through selinux_[sem,msg_queue,shm]_free_security(), we can race if the structure is freed before other tasks are done with it, creating a use-after-free condition. Manfred illustrates this nicely, for instance with shared mem and selinux:
->
do_shmatcallsrcu_read_lock()
->do_shmatcallsshm_object_check().
Checks that the object is still valid - but doesn't acquire any locks. Then it returns.
->do_shmatcallssecurity_shm_shmat(e.g.selinux_shm_shmat) ->selinux_shm_shmatcallsipc_has_perm()
->ipc_has_permaccessesipc_perms->security
shm_close()
->shm_closeacquiresrw_mutex&shm_lock
->shm_closecallsshm_destroy
->shm_destroycallssecurity_shm_free(e.g.selinux_shm_free_security) ->selinux_shm_free_securitycallsipc_free_security(&shp->shm_perm)->ipc_free_securitycallskfree(ipc_perms->security)This patch delays the freeing of the security structures after all RCU readers are done. Furthermore it aligns the security life cycle with that of the rest of IPC - freeing them based on the reference counter. For situations where we need not free security, the current behavior is kept. Linus states:
"... the old behavior was suspect for another reason too: having the
security blob go away from under a user sounds like it could cause various other problems anyway, so I think the old code was at least _prone_ to bugs even if it didn't have catastrophic behavior."I have tested this patch with IPC testcases from LTP on both my quad-core laptop and on a 64 core NUMA server. In both cases selinux is enabled, and tests pass for both voluntary and forced preemption models. While the mentioned races are theoretical (at least no one as reported them), I wanted to make sure that this new logic doesn't break anything we weren't aware of.
Quoted 5 lines
Additionally, Manfred's concerns about the grace period are quite valid, and it obviously assumes that the ->security nil dereference issue still exists to some extent. Changes in RCU are something to consider as well. This is all pretty iffy though, lets make sure we are looking at the right kernel first.
Thanks,
Ethan
Quoted 4 lines
Thanks,
Davidlohr
Stephen,
Quoted 43 lines · Stephen Smalley
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
sys_semget() ->newary() ->security_sem_alloc() ->sem_alloc_security() selinux_sem_alloc_security() ->ipc_alloc_security() { ->rc = avc_has_perm() if (rc) {ipc_free_security(&sma->sem_perm); return rc;Quoted 8 lines
Quoted 6 lines
We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an
ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never callsipc_addid()and the semaphore set is not created. So how then can you callsemtimedop()on it?Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4?That was my first guess when I read the bug report - but it can't be the fix, because
security_sem_alloc()is before theipc_addid(), with or without the patch.thread A:
thread B:
semtimedop()
->sem_obtain_object_check()semctl(IPC_RMID) -> freeary() -> ipc_rcu_putref() -> call_rcu() -> somehow a grace period -> sem_rcu_free() -> security_sem_free()
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Perhaps: modify
ipc_free_security()to hexdump perm and a few more bytes if the pointer is NULL?I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.
You said the patch was tested with 3.19-rc5.
I just threw the 3.19-rc5 with my test patch to the 'user', he said he doesn't hit. maybe he didn't hit or occasionally failed to reproduce it.
Quoted 2 lines
But did you reproduce
the bug on that kernel version before the patch?
Good news is not hit yet.
Quoted 2 lines
If not, what kernel
version were you running when you triggered the bug?
To be honest, a kernel from distro, but not released, but before we
get it clear, we wouldn't public more.
Thanks,
Ethan
Quoted 4 lines · ethan zhao
Quoted 2 lines
If not, what kernel
version were you running when you triggered the bug?To be honest, a kernel from distro, but not released, but before we
get it clear, we wouldn't public more.
Sheesh, could Oracle be any more (ridiculously) secretive about what the hell kernel(s) they run... it's like pulling teeth. *sigh*
Davidlohr,
I read your commit 53dad6d3a8e5ac1af8bacc6ac2134ae1a8b085f1,
ipc: fix race with LSMs
The issue we hit without the above patch, the race may happen when
process call semctl with IPC_RMID just as Manfred Spraul mentioned:
Thread A:
IPC_RMID
-> freeary()
->wake_up_sem_queue_do()
Thread B:
->security_sem_free() semtimedop()->ipcperms()
->ipc_rcu_putref()
If this is the only race, the bug should be fixed with your patch applied
(not verified yet on my case).
Thanks,
Ethan
Quoted 9 lines · Davidlohr Bueso
Quoted 4 lines · ethan zhao
Quoted 2 lines
If not, what kernel
version were you running when you triggered the bug?To be honest, a kernel from distro, but not released, but before we
get it clear, we wouldn't public more.Sheesh, could Oracle be any more (ridiculously) secretive about what the hell kernel(s) they run... it's like pulling teeth. *sigh*