Skip to contentSkip to conversation

Selinux/hooks.c: Fix a NULL pointer dereference caused by semop()

PATCH In review Ethan Zhao 7 participants 18 messages Jan 20, 2015 – Jan 23, 2015 SELinux security module

lore
Share & tools
lore

Ethan Zhao

author
#0 lore raw

PATCH 1/1 Selinux/hooks.c: Fix a NULL pointer dereference caused by semop()

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its return value in process opcmon. and semget() failed to check permission in function avc_has_perm() then sem_perm->security was freed shown as following:

     sys_semget()
     ->newary()
	   ->security_sem_alloc()
	     ->sem_alloc_security()
		   selinux_sem_alloc_security()
		   ->ipc_alloc_security() {
		     ->rc = avc_has_perm()
				if (rc) {
					ipc_free_security(&sma->sem_perm);
					return rc;

     So ipc_perms->security was NULL, then semtimedop() was called as
     following:

	  sys_semtimedop() / semop()
	  ->selinux_sem_semop()
	   ->ipc_has_perm()
	     ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);

^- NULL pointer dereference happens

The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.

---
 security/selinux/hooks.c | 2 ++
 1 file changed, 2 insertions(+)
security/selinux/hooks.c +2 -0
index 6da7532..bbe76f5 100644
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,
 	u32 sid = current_sid();
 
 	isec = ipc_perms->security;
+	if (!isec)
+		return -EACCES;
 
 	ad.type = LSM_AUDIT_DATA_IPC;
 	ad.u.ipc_id = ipc_perms->key;
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 43 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its return value in process opcmon. and semget() failed to check permission in function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
     sys_semget()
     ->newary()
	   ->security_sem_alloc()
	     ->sem_alloc_security()
		   selinux_sem_alloc_security()
		   ->ipc_alloc_security() {
		     ->rc = avc_has_perm()
				if (rc) {
					ipc_free_security(&sma->sem_perm);
					return rc;

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 24 lines
     So ipc_perms->security was NULL, then semtimedop() was called as
     following:

	  sys_semtimedop() / semop()
	  ->selinux_sem_semop()
	   ->ipc_has_perm()
	     ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);

^- NULL pointer dereference happens

The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.

---
 security/selinux/hooks.c | 2 ++
 1 file changed, 2 insertions(+)
security/selinux/hooks.c +0 -0
index 6da7532..bbe76f5 100644

security/selinux/hooks.c static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, · line 5136

Quoted 8 lines
security/selinux/hooks.c +2 -0
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,
 	u32 sid = current_sid();
 
 	isec = ipc_perms->security;
+	if (!isec)
+		return -EACCES;
 
 	ad.type = LSM_AUDIT_DATA_IPC;
 	ad.u.ipc_id = ipc_perms->key;

That is not the correct fix; it just hides a bug. If we reach ipc_has_perm() with a NULL isec, it is a bug in the ipc code.

#2 lore raw

Hi,

Quoted 54 lines · Stephen Smalley
Quoted 43 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its return value in process opcmon. and semget() failed to check permission in function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
      sys_semget()
      ->newary()
	   ->security_sem_alloc()
	     ->sem_alloc_security()
		   selinux_sem_alloc_security()
		   ->ipc_alloc_security() {
		     ->rc = avc_has_perm()
				if (rc) {
					ipc_free_security(&sma->sem_perm);
					return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

My only idea would be a race of semtimedop() with IPC_RMID:
If a rcu grace period happens between sem_obtain_object_check() and the ipc_has_perm() call, the the observed NULL pointer assignment would happen.

Quoted 10 lines
Quoted 10 lines
      So ipc_perms->security was NULL, then semtimedop() was called as
      following:

	  sys_semtimedop() / semop()
	  ->selinux_sem_semop()
	   ->ipc_has_perm()
	     ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);

^- NULL pointer dereference happens

The test kernel was running on VMware.

Can you give some more details?
- which RCU type?
- PREEMPT?

--
Manfred

#3 lore raw
Quoted 67 lines · Manfred Spraul

Hi,

Quoted 58 lines · Stephen Smalley
Quoted 47 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8
<0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its
return value in process opcmon. and semget() failed to check permission in
function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
      sys_semget()
      ->newary()
       ->security_sem_alloc()
         ->sem_alloc_security()
           selinux_sem_alloc_security()
           ->ipc_alloc_security() {
             ->rc = avc_has_perm()
                if (rc) {
                    ipc_free_security(&sma->sem_perm);
                    return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 3 lines

My only idea would be a race of semtimedop() with IPC_RMID:
If a rcu grace period happens between sem_obtain_object_check() and the ipc_has_perm() call, the the observed NULL pointer assignment would happen.

We only free and clear the ipc_perms->security field on a failure during newary() -> security_sem_alloc(), in which case we fail with an error before the ipc_addid() call has occurred, or during sem_rcu_free() ->
security_sem_free() just prior to calling ipc_rcu_free(). So I don't
see how ipc_perms->security can be NULL in ipc_has_perm(). We could rcu free the ipc_perms->security field but I don't see why that would be correct/necessary.

#4 lore raw

What kernel version was this? Didn't we have this problem and solve it upstream some time ago? IPC could be allocated with a valid security context, the ipc would be freed. the isec was free'd syncronously, but then the ipc could stick around until some rcu period or some usage flag got to 0, then it got freed...

this seems so familiar, but it was a while ago

Quoted 80 lines · Stephen Smalley
Quoted 67 lines · Manfred Spraul

Hi,

Quoted 58 lines · Stephen Smalley
Quoted 47 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8
<0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its
return value in process opcmon. and semget() failed to check permission in
function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
      sys_semget()
      ->newary()
       ->security_sem_alloc()
         ->sem_alloc_security()
           selinux_sem_alloc_security()
           ->ipc_alloc_security() {
             ->rc = avc_has_perm()
                if (rc) {
                    ipc_free_security(&sma->sem_perm);
                    return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 3 lines

My only idea would be a race of semtimedop() with IPC_RMID:
If a rcu grace period happens between sem_obtain_object_check() and the ipc_has_perm() call, the the observed NULL pointer assignment would happen.

We only free and clear the ipc_perms->security field on a failure during newary() -> security_sem_alloc(), in which case we fail with an error before the ipc_addid() call has occurred, or during sem_rcu_free() ->
security_sem_free() just prior to calling ipc_rcu_free(). So I don't
see how ipc_perms->security can be NULL in ipc_has_perm(). We could rcu free the ipc_perms->security field but I don't see why that would be correct/necessary.

#5 lore raw
Quoted 7 lines · Eric Paris

What kernel version was this? Didn't we have this problem and solve it upstream some time ago? IPC could be allocated with a valid security context, the ipc would be freed. the isec was free'd syncronously, but then the ipc could stick around until some rcu period or some usage flag got to 0, then it got freed...

this seems so familiar, but it was a while ago

Don't know the kernel version.

Are you thinking of the inode security bug? Because in that case the vfs changed underneath us to rcu free the inode but left the security_inode_free() call outside of the rcu callback (due to the rcu callback only being used in the default case where the filesystem does not implement its own destroy_inode() method). In this case, we don't seem to have that situation AFAICS.

Quoted 85 lines · Stephen Smalley
Quoted 80 lines · Manfred Spraul
Quoted 67 lines

Hi,

Quoted 58 lines · Stephen Smalley
Quoted 47 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8
<0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its
return value in process opcmon. and semget() failed to check permission in
function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
      sys_semget()
      ->newary()
       ->security_sem_alloc()
         ->sem_alloc_security()
           selinux_sem_alloc_security()
           ->ipc_alloc_security() {
             ->rc = avc_has_perm()
                if (rc) {
                    ipc_free_security(&sma->sem_perm);
                    return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 3 lines

My only idea would be a race of semtimedop() with IPC_RMID:
If a rcu grace period happens between sem_obtain_object_check() and the ipc_has_perm() call, the the observed NULL pointer assignment would happen.

We only free and clear the ipc_perms->security field on a failure during newary() -> security_sem_alloc(), in which case we fail with an error before the ipc_addid() call has occurred, or during sem_rcu_free() ->
security_sem_free() just prior to calling ipc_rcu_free(). So I don't
see how ipc_perms->security can be NULL in ipc_has_perm(). We could rcu free the ipc_perms->security field but I don't see why that would be correct/necessary.

author
#6 lore raw
In reply to Manfred Spraul

Stephen,

Quoted 76 lines · Manfred Spraul

Hi,

Quoted 54 lines · Stephen Smalley
Quoted 43 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its return value in process opcmon. and semget() failed to check permission in function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
      sys_semget()
      ->newary()
       ->security_sem_alloc()
         ->sem_alloc_security()
           selinux_sem_alloc_security()
           ->ipc_alloc_security() {
             ->rc = avc_has_perm()
                if (rc) {
                    ipc_free_security(&sma->sem_perm);
                    return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

My only idea would be a race of semtimedop() with IPC_RMID:
If a rcu grace period happens between sem_obtain_object_check() and the ipc_has_perm() call, the the observed NULL pointer assignment would happen.

Quoted 10 lines
Quoted 10 lines
      So ipc_perms->security was NULL, then semtimedop() was called as
      following:

      sys_semtimedop() / semop()
      ->selinux_sem_semop()
       ->ipc_has_perm()
         ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);
                                           ^- NULL pointer 

dereference happens

Quoted 4 lines
Quoted 1 line
Quoted 1 line

The test kernel was running on VMware.

Can you give some more details?
- which RCU type?
- PREEMPT?

Let me figure out more detail about it and share with you.

Thanks,
Ethan

Quoted 3 lines
Signature · 2 lines
-- 
    Manfred
#7 lore raw
In reply to Stephen Smalley
Quoted 55 lines · Stephen Smalley
Quoted 43 lines · Ethan Zhao

A NULL pointer dereference was observed as following panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
...
Process opcmon (pid: 30712, threadinfo ffff880237f2a000,
task ffff88022ac70e40)
Stack:

ffff880237f2bc04 ffffffff01020953 ffff880237f2bce8 ffffffff8125818e
0000000000000001 0000000037f78004 ffff880237f2bcd8 ffffffff81273619
ffff880237f2bce8 ffffffff8126e3e6 ffff880237f2bf68 ffffffff8125c206

Call Trace:
[<ffffffff8125818e>] ? ipcperms+0xae/0x110
[<ffffffff81273619>] selinux_sem_semop+0x19/0x20
[<ffffffff8126e3e6>] security_sem_semop+0x16/0x20
[<ffffffff8125c206>] sys_semtimedop+0x346/0x750
[<ffffffff81188c0c>] ? handle_pte_fault+0x1dc/0x200
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff8109a763>] ? up_read+0x23/0x40
[<ffffffff8161d830>] ? __do_page_fault+0x280/0x500
[<ffffffff81182f1c>] ? might_fault+0x5c/0xb0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff810d97d0>] ? __lock_release+0x90/0x1b0
[<ffffffff81081f96>] ? sys_newuname+0x66/0xf0
[<ffffffff81622f45>] ? sysret_check+0x22/0x5d
[<ffffffff8125c620>] sys_semop+0x10/0x20
[<ffffffff81622f19>] system_call_fastpath+0x16/0x1b
Code: b8 00 00 48 8b 80 48 06 00 00 41 8b 54 24 40 4c 8d

45 d0 89 d9 45 31 c9 48 8b 40 70 8b 78 04 49 8b 44 24 60 c6 45 d0 04 89 55 d8 <0f> b7 10 8b 70 04 e8 0a dc ff ff 48 83 c4 20 5b 41 5c c9 c3 90 RIP [<ffffffff812735eb>] ipc_has_perm+0x4b/0x60
RSP <ffff880237f2bc98>
CR2: 0000000000000000

The root cause is semtimedop() was called after semget() without checking its return value in process opcmon. and semget() failed to check permission in function avc_has_perm() then sem_perm->security was freed shown as following:

Quoted 10 lines
     sys_semget()
     ->newary()
         ->security_sem_alloc()
           ->sem_alloc_security()
                 selinux_sem_alloc_security()
                 ->ipc_alloc_security() {
                   ->rc = avc_has_perm()
                              if (rc) {
                                      ipc_free_security(&sma->sem_perm);
                                      return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Seems it wouldn't happen after commit e8577d1f0329d4842e8302e289fb2c22156abef4 ?

Thanks,
Ethan

Quoted 42 lines
Quoted 25 lines
     So ipc_perms->security was NULL, then semtimedop() was called as
     following:

        sys_semtimedop() / semop()
        ->selinux_sem_semop()
         ->ipc_has_perm()
           ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);
                                          ^- NULL pointer dereference happens

The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.

---
 security/selinux/hooks.c | 2 ++
 1 file changed, 2 insertions(+)
security/selinux/hooks.c +0 -0
index 6da7532..bbe76f5 100644

security/selinux/hooks.c static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, · line 5136

Quoted 8 lines
security/selinux/hooks.c +2 -0
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,
      u32 sid = current_sid();
 
      isec = ipc_perms->security;
+     if (!isec)
+             return -EACCES;
 
      ad.type = LSM_AUDIT_DATA_IPC;
      ad.u.ipc_id = ipc_perms->key;

That is not the correct fix; it just hides a bug. If we reach ipc_has_perm() with a NULL isec, it is a bug in the ipc code.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#8 lore raw
Quoted 12 lines · Ethan Zhao
Quoted 11 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
      sys_semget()
      ->newary()
          ->security_sem_alloc()
            ->sem_alloc_security()
                  selinux_sem_alloc_security()
                  ->ipc_alloc_security() {
                    ->rc = avc_has_perm()
                               if (rc) {
                                       ipc_free_security(&sma->sem_perm);
                                       return rc;
Quoted 7 lines
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Seems it wouldn't happen after commit e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

            semctl(IPC_RMID)
            -> freeary()
            -> ipc_rcu_putref()
            -> call_rcu()
-> somehow a grace period
            -> sem_rcu_free()
            -> security_sem_free()

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

--
Manfred

Quoted 42 lines

Thanks,
Ethan

Quoted 40 lines
Quoted 24 lines
      So ipc_perms->security was NULL, then semtimedop() was called as
      following:

         sys_semtimedop() / semop()
         ->selinux_sem_semop()
          ->ipc_has_perm()
            ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);
                                           ^- NULL pointer dereference happens

The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space. This patch was tested with v3.19-rc5.

---
  security/selinux/hooks.c | 2 ++
  1 file changed, 2 insertions(+)
security/selinux/hooks.c +0 -0
index 6da7532..bbe76f5 100644

security/selinux/hooks.c static int ipc_has_perm(struct kern_ipc_perm *ipc_perms, · line 5136

Quoted 8 lines
security/selinux/hooks.c +2 -0
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,
       u32 sid = current_sid();
 
       isec = ipc_perms->security;
+     if (!isec)
+             return -EACCES;
 
       ad.type = LSM_AUDIT_DATA_IPC;
       ad.u.ipc_id = ipc_perms->key;

That is not the correct fix; it just hides a bug. If we reach ipc_has_perm() with a NULL isec, it is a bug in the ipc code.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#9 lore raw
Quoted 46 lines · Manfred Spraul
Quoted 25 lines · Ethan Zhao
Quoted 15 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
      sys_semget()
      ->newary()
          ->security_sem_alloc()
            ->sem_alloc_security()
                  selinux_sem_alloc_security()
                  ->ipc_alloc_security() {
                    ->rc = avc_has_perm()
                               if (rc) {
Quoted 2 lines
ipc_free_security(&sma->sem_perm);
                                       return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 2 lines

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

            semctl(IPC_RMID)
            -> freeary()
            -> ipc_rcu_putref()
            -> call_rcu()
-> somehow a grace period
            -> sem_rcu_free()
            -> security_sem_free()
Quoted 2 lines

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

Thanks,
Ethan

Quoted 54 lines

--
Manfred

Quoted 47 lines

Thanks,
Ethan

Quoted 45 lines
Quoted 34 lines
      So ipc_perms->security was NULL, then semtimedop() was called as
      following:

         sys_semtimedop() / semop()
         ->selinux_sem_semop()
          ->ipc_has_perm()
            ->avc_has_perm(sid, isec->sid, isec->sclass, perms, &ad);
                                           ^- NULL pointer dereference

happens

The test kernel was running on VMware.
This patch use to fix this serious security issue could be triggered by user space.
This patch was tested with v3.19-rc5.

---
  security/selinux/hooks.c | 2 ++
  1 file changed, 2 insertions(+)
security/selinux/hooks.c +2 -0
index 6da7532..bbe76f5 100644
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm
 *ipc_perms,
       u32 sid = current_sid();
 
       isec = ipc_perms->security;
+     if (!isec)
+             return -EACCES;
 
       ad.type = LSM_AUDIT_DATA_IPC;
Quoted 2 lines
       ad.u.ipc_id = ipc_perms->key;

That is not the correct fix; it just hides a bug. If we reach ipc_has_perm() with a NULL isec, it is a bug in the ipc code.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#10 lore raw
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
       sys_semget()
       ->newary()
           ->security_sem_alloc()
             ->sem_alloc_security()
                   selinux_sem_alloc_security()
                   ->ipc_alloc_security() {
                     ->rc = avc_has_perm()
                                if (rc) {
ipc_free_security(&sma->sem_perm);
                                        return rc;
Quoted 8 lines
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

             semctl(IPC_RMID)
             -> freeary()
             -> ipc_rcu_putref()
             -> call_rcu()
-> somehow a grace period
             -> sem_rcu_free()
             -> security_sem_free()
Quoted 4 lines
Quoted 2 lines

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

Is the issue reproducable?
If yes, can you try something like the attached patch?

Quoted 11 lines
security/selinux/hooks.c +2 -0
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm
 *ipc_perms,
        u32 sid = current_sid();
 
        isec = ipc_perms->security;
+     if (!isec)
+             return -EACCES;
 
        ad.type = LSM_AUDIT_DATA_IPC;
Quoted 1 line
        ad.u.ipc_id = ipc_perms->key;

ipc_has_perm() runs without any spinlocks/semaphores, only rcu_read_lock().
Testing for ipc_perms->security!=NULL does solve the issue that ipc_perm->key could be an access to kfree'd memory: Nothing prevents that the kfree could happen just after the test.

I.e.: The patch can't be the right solution.

--
Manfred

ipc/sem.c +6 -0
index 6115146..80371dc 100644
@@ -248,6 +248,7 @@ static void sem_rcu_free(struct rcu_head *head)
 	struct ipc_rcu *p = container_of(head, struct ipc_rcu, rcu);
 	struct sem_array *sma = ipc_rcu_to_struct(p);
 
+pr_info("sem_rcu_free: sma %p\n",sma);
 	security_sem_free(sma);
 	ipc_rcu_free(head);
 }
@@ -529,6 +530,7 @@ static int newary(struct ipc_namespace *ns, struct ipc_params *params)
 	sma->sem_nsems = nsems;
 	sma->sem_ctime = get_seconds();
 
+pr_info("newary: sma %p becomes visible\n",sma);
 	id = ipc_addid(&sem_ids(ns), &sma->sem_perm, ns->sc_semmni);
 	if (id < 0) {
 		ipc_rcu_putref(sma, sem_rcu_free);
@@ -1118,6 +1120,7 @@ static void freeary(struct ipc_namespace *ns, struct kern_ipc_perm *ipcp)
 
 	/* Remove the semaphore set from the IDR */
 	sem_rmid(ns, sma);
+pr_info("freeary: sma %p unlinked\n",sma);
 	sem_unlock(sma, -1);
 	rcu_read_unlock();
 
@@ -1860,6 +1863,9 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
 	if (ipcperms(ns, &sma->sem_perm, alter ? S_IWUGO : S_IRUGO))
 		goto out_rcu_wakeup;
 
+	if (sma->sem_perm.security == NULL) {
+		pr_info("sma %p: sem_perm.security == NULL\n", sma);
+	}
 	error = security_sem_semop(sma, sops, nsops, alter);
 	if (error)
 		goto out_rcu_wakeup;
security/selinux/hooks.c +8 -0
index 6da7532..1499787 100644
@@ -5088,6 +5088,7 @@ static int ipc_alloc_security(struct task_struct *task,
 	isec->sclass = sclass;
 	isec->sid = sid;
 	perm->security = isec;
+pr_info("ipc_alloc_security for perm %p.\n", perm);
 
 	return 0;
 }
@@ -5096,6 +5097,7 @@ static void ipc_free_security(struct kern_ipc_perm *perm)
 {
 	struct ipc_security_struct *isec = perm->security;
 	perm->security = NULL;
+pr_info("ipc_free_security for perm %p.\n", perm);
 	kfree(isec);
 }
 
@@ -5129,6 +5131,12 @@ static int ipc_has_perm(struct kern_ipc_perm *ipc_perms,
 	u32 sid = current_sid();
 
 	isec = ipc_perms->security;
+	if (isec == NULL) {
+		struct sem_array *sma = container_of(ipc_perms, struct sem_array, sem_perm);
+
+		pr_info("sma %p, sem_base %p, deleted %d with NULL isec\n",
+					sma, sma->sem_base, sma->sem_perm.deleted);
+	}
 
 	ad.type = LSM_AUDIT_DATA_IPC;
 	ad.u.ipc_id = ipc_perms->key;
author
#11 lore raw

Manfred,

Quoted 43 lines · Manfred Spraul
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
       sys_semget()
       ->newary()
           ->security_sem_alloc()
             ->sem_alloc_security()
                   selinux_sem_alloc_security()
                   ->ipc_alloc_security() {
                     ->rc = avc_has_perm()
                                if (rc) {
ipc_free_security(&sma->sem_perm);
                                        return rc;
Quoted 8 lines
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

             semctl(IPC_RMID)
             -> freeary()
             -> ipc_rcu_putref()
             -> call_rcu()
-> somehow a grace period
             -> sem_rcu_free()
             -> security_sem_free()
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

Is the issue reproducable?

It was hit on an user's VMware while running a process named "opcmon"
maybe ported from HP_UX. I asked for the vmwcore, but not get it yet.

Quoted 1 line

If yes, can you try something like the attached patch?

Yes, will.

Thanks,
Ethan

Quoted 20 lines
Quoted 12 lines
security/selinux/hooks.c +2 -0
@@ -5129,6 +5129,8 @@ static int ipc_has_perm(struct kern_ipc_perm
 *ipc_perms,
        u32 sid = current_sid();
 
        isec = ipc_perms->security;
+     if (!isec)
+             return -EACCES;
 
        ad.type = LSM_AUDIT_DATA_IPC;
Quoted 1 line
        ad.u.ipc_id = ipc_perms->key;

ipc_has_perm() runs without any spinlocks/semaphores, only rcu_read_lock().
Testing for ipc_perms->security!=NULL does solve the issue that ipc_perm->key could be an access to kfree'd memory: Nothing prevents that the kfree could happen just after the test.

I.e.: The patch can't be the right solution.

Signature · 1 line
    Manfred
#12 lore raw
In reply to Ethan Zhao
Quoted 48 lines · Ethan Zhao
Quoted 46 lines · Manfred Spraul
Quoted 25 lines · Ethan Zhao
Quoted 15 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
      sys_semget()
      ->newary()
          ->security_sem_alloc()
            ->sem_alloc_security()
                  selinux_sem_alloc_security()
                  ->ipc_alloc_security() {
                    ->rc = avc_has_perm()
                               if (rc) {
Quoted 2 lines
ipc_free_security(&sma->sem_perm);
                                       return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 2 lines

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

            semctl(IPC_RMID)
            -> freeary()
            -> ipc_rcu_putref()
            -> call_rcu()
-> somehow a grace period
            -> sem_rcu_free()
            -> security_sem_free()
Quoted 5 lines
Quoted 2 lines

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

You said the patch was tested with 3.19-rc5. But did you reproduce the bug on that kernel version before the patch? If not, what kernel version were you running when you triggered the bug?

#13 lore raw
Quoted 52 lines · Stephen Smalley
Quoted 51 lines · Ethan Zhao
Quoted 48 lines · Manfred Spraul
Quoted 25 lines · Ethan Zhao
Quoted 15 lines · Stephen Smalley
Quoted 10 lines · Ethan Zhao
      sys_semget()
      ->newary()
          ->security_sem_alloc()
            ->sem_alloc_security()
                  selinux_sem_alloc_security()
                  ->ipc_alloc_security() {
                    ->rc = avc_has_perm()
                               if (rc) {
Quoted 2 lines
ipc_free_security(&sma->sem_perm);
                                       return rc;
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Quoted 2 lines

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

            semctl(IPC_RMID)
            -> freeary()
            -> ipc_rcu_putref()
            -> call_rcu()
-> somehow a grace period
            -> sem_rcu_free()
            -> security_sem_free()

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

Quoted 6 lines
Quoted 2 lines

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

You said the patch was tested with 3.19-rc5. But did you reproduce the bug on that kernel version before the patch? If not, what kernel version were you running when you triggered the bug?

Also, is this a vanilla kernel? Or from a distro?

Essentially, did the kernel with the reproducible bug have:

commit 53dad6d3a8e5ac1af8bacc6ac2134ae1a8b085f1

Author: Davidlohr Bueso <[email protected]>
Date:   Mon Sep 23 17:04:45 2013 -0700

ipc: fix race with LSMs

Currently, IPC mechanisms do security and auditing related checks under RCU. However, since security modules can free the security structure, for example, through selinux_[sem,msg_queue,shm]_free_security(), we can race if the structure is freed before other tasks are done with it, creating a use-after-free condition. Manfred illustrates this nicely, for instance with shared mem and selinux:

-> do_shmat calls rcu_read_lock()
-> do_shmat calls shm_object_check().
Checks that the object is still valid - but doesn't acquire any locks. Then it returns.
-> do_shmat calls security_shm_shmat (e.g. selinux_shm_shmat) -> selinux_shm_shmat calls ipc_has_perm()
-> ipc_has_perm accesses ipc_perms->security

shm_close()
-> shm_close acquires rw_mutex & shm_lock
-> shm_close calls shm_destroy
-> shm_destroy calls security_shm_free (e.g. selinux_shm_free_security) -> selinux_shm_free_security calls ipc_free_security(&shp->shm_perm) -> ipc_free_security calls kfree(ipc_perms->security)

This patch delays the freeing of the security structures after all RCU readers are done. Furthermore it aligns the security life cycle with that of the rest of IPC - freeing them based on the reference counter. For situations where we need not free security, the current behavior is kept. Linus states:

"... the old behavior was suspect for another reason too: having the
security blob go away from under a user sounds like it could cause various other problems anyway, so I think the old code was at least _prone_ to bugs even if it didn't have catastrophic behavior."

I have tested this patch with IPC testcases from LTP on both my quad-core laptop and on a 64 core NUMA server. In both cases selinux is enabled, and tests pass for both voluntary and forced preemption models. While the mentioned races are theoretical (at least no one as reported them), I wanted to make sure that this new logic doesn't break anything we weren't aware of.

Additionally, Manfred's concerns about the grace period are quite valid, and it obviously assumes that the ->security nil dereference issue still exists to some extent. Changes in RCU are something to consider as well. This is all pretty iffy though, lets make sure we are looking at the right kernel first.

Thanks,
Davidlohr

author
#14 lore raw

Davidlohr,

Quoted 44 lines · Davidlohr Bueso
Quoted 43 lines · Stephen Smalley
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
       sys_semget()
       ->newary()
           ->security_sem_alloc()
             ->sem_alloc_security()
                   selinux_sem_alloc_security()
                   ->ipc_alloc_security() {
                     ->rc = avc_has_perm()
                                if (rc) {
ipc_free_security(&sma->sem_perm);
                                        return rc;
Quoted 8 lines
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

             semctl(IPC_RMID)
             -> freeary()
             -> ipc_rcu_putref()
             -> call_rcu()
-> somehow a grace period
             -> sem_rcu_free()
             -> security_sem_free()
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

You said the patch was tested with 3.19-rc5. But did you reproduce the bug on that kernel version before the patch? If not, what kernel version were you running when you triggered the bug?

Also, is this a vanilla kernel? Or from a distro?

The hard thing, it is hit on customer's environment, the issue kernel
doesn't
have many commits far from the last about IPC/SElinux.

Quoted 1 line

Essentially, did the kernel with the reproducible bug have:

Not easy to do reproduce it is triggered by a process "opcmon" not
public to everyone.
What I have is the panic log. the vmware not get yet.

Thanks,
Ethan

Quoted 5 lines

commit 53dad6d3a8e5ac1af8bacc6ac2134ae1a8b085f1

Author: Davidlohr Bueso <[email protected]>
Date:   Mon Sep 23 17:04:45 2013 -0700

ipc: fix race with LSMs

No, the kernel doesn't have this commit, will try it.

Quoted 42 lines

Currently, IPC mechanisms do security and auditing related checks under RCU. However, since security modules can free the security structure, for example, through selinux_[sem,msg_queue,shm]_free_security(), we can race if the structure is freed before other tasks are done with it, creating a use-after-free condition. Manfred illustrates this nicely, for instance with shared mem and selinux:

-> do_shmat calls rcu_read_lock()
-> do_shmat calls shm_object_check().
Checks that the object is still valid - but doesn't acquire any locks. Then it returns.
-> do_shmat calls security_shm_shmat (e.g. selinux_shm_shmat) -> selinux_shm_shmat calls ipc_has_perm()
-> ipc_has_perm accesses ipc_perms->security

shm_close()
-> shm_close acquires rw_mutex & shm_lock
-> shm_close calls shm_destroy
-> shm_destroy calls security_shm_free (e.g. selinux_shm_free_security) -> selinux_shm_free_security calls ipc_free_security(&shp->shm_perm) -> ipc_free_security calls kfree(ipc_perms->security)

This patch delays the freeing of the security structures after all RCU readers are done. Furthermore it aligns the security life cycle with that of the rest of IPC - freeing them based on the reference counter. For situations where we need not free security, the current behavior is kept. Linus states:

"... the old behavior was suspect for another reason too: having the
security blob go away from under a user sounds like it could cause various other problems anyway, so I think the old code was at least _prone_ to bugs even if it didn't have catastrophic behavior."

I have tested this patch with IPC testcases from LTP on both my quad-core laptop and on a 64 core NUMA server. In both cases selinux is enabled, and tests pass for both voluntary and forced preemption models. While the mentioned races are theoretical (at least no one as reported them), I wanted to make sure that this new logic doesn't break anything we weren't aware of.

Quoted 5 lines

Additionally, Manfred's concerns about the grace period are quite valid, and it obviously assumes that the ->security nil dereference issue still exists to some extent. Changes in RCU are something to consider as well. This is all pretty iffy though, lets make sure we are looking at the right kernel first.

Thanks,
Ethan

Quoted 4 lines

Thanks,
Davidlohr

author
#15 lore raw
In reply to Stephen Smalley

Stephen,

Quoted 43 lines · Stephen Smalley
Quoted 42 lines · Ethan Zhao
Quoted 40 lines · Manfred Spraul
Quoted 14 lines · Ethan Zhao
Quoted 12 lines · Stephen Smalley
Quoted 11 lines · Ethan Zhao
       sys_semget()
       ->newary()
           ->security_sem_alloc()
             ->sem_alloc_security()
                   selinux_sem_alloc_security()
                   ->ipc_alloc_security() {
                     ->rc = avc_has_perm()
                                if (rc) {
ipc_free_security(&sma->sem_perm);
                                        return rc;
Quoted 8 lines
Quoted 6 lines

We free the security structure here to avoid a memory leak on a failed/denied semaphore set creation. In this situation, we return an error to the caller (ultimately to newary), it does an ipc_rcu_putref(sma, ipc_rcu_free), and it returns an error to the caller. Thus, it never calls ipc_addid() and the semaphore set is not created. So how then can you call semtimedop() on it?

Seems it wouldn't happen after commit
e8577d1f0329d4842e8302e289fb2c22156abef4 ?

That was my first guess when I read the bug report - but it can't be the fix, because security_sem_alloc() is before the ipc_addid(), with or without the patch.

thread A:
thread B:

semtimedop()
-> sem_obtain_object_check()

             semctl(IPC_RMID)
             -> freeary()
             -> ipc_rcu_putref()
             -> call_rcu()
-> somehow a grace period
             -> sem_rcu_free()
             -> security_sem_free()
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Perhaps: modify ipc_free_security() to hexdump perm and a few more bytes if the pointer is NULL?

I tried to ask for vmcore and do more analysis, basically, the race condition still exists and open a hole to be DoS.

You said the patch was tested with 3.19-rc5.

I just threw the 3.19-rc5 with my test patch to the 'user', he said he doesn't hit. maybe he didn't hit or occasionally failed to reproduce it.

Quoted 2 lines

But did you reproduce
the bug on that kernel version before the patch?

Good news is not hit yet.

Quoted 2 lines

If not, what kernel
version were you running when you triggered the bug?

To be honest, a kernel from distro, but not released, but before we
get it clear, we wouldn't public more.

Thanks,
Ethan

#16 lore raw
Quoted 4 lines · ethan zhao
Quoted 2 lines

If not, what kernel
version were you running when you triggered the bug?

To be honest, a kernel from distro, but not released, but before we
get it clear, we wouldn't public more.

Sheesh, could Oracle be any more (ridiculously) secretive about what the hell kernel(s) they run... it's like pulling teeth. *sigh*

#17 lore raw

Davidlohr,

I read your commit 53dad6d3a8e5ac1af8bacc6ac2134ae1a8b085f1,
ipc: fix race with LSMs

The issue we hit without the above patch, the race may happen when
process call semctl with IPC_RMID just as Manfred Spraul mentioned:

Thread A:
IPC_RMID
-> freeary()

       ->wake_up_sem_queue_do()
                                                                   Thread B:
       ->security_sem_free()                           semtimedop()

->ipcperms()

       ->ipc_rcu_putref()

If this is the only race, the bug should be fixed with your patch applied
(not verified yet on my case).

Thanks,
Ethan

Quoted 9 lines · Davidlohr Bueso
Quoted 4 lines · ethan zhao
Quoted 2 lines

If not, what kernel
version were you running when you triggered the bug?

To be honest, a kernel from distro, but not released, but before we
get it clear, we wouldn't public more.

Sheesh, could Oracle be any more (ridiculously) secretive about what the hell kernel(s) they run... it's like pulling teeth. *sigh*