Skip to contentSkip to conversation

EACCES errors using preload libs

DISCUSSION Steve Wise 2 participants 5 messages Sep 30, 2016 2 messages not in the archive InfiniBand

lore
Share & tools
lore

Steve Wise

author
#0 lore raw

Hey Jason, I'm looking into a problem that appears to have been introduced by:

commit e6bd18f57aad1a2d1ef40e646d03ed0f2515c9e3

Author: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Date:   Sun Apr 10 19:13:13 2016 -0600

IB/security: Restrict use of the write() interface

I'm using a preload library (similar to rsockets) that enables a user-mode UDP stack over chelsio hardware. The preload lib uses user rdma verbs calls. In some cases, verbs calls are getting EACCES. Can you explain what "suspicious" write API calls this is disallowing:

static inline bool ib_safe_file_access(struct file *filp)

{
        return filp->f_cred == current_cred() && segment_eq(get_fs(),

USER_DS);
}

The single case I see failing so far is trying to preload my library with netserver. If I run netserver with '-D -f' which avoids fork() calls, it works fine. If I omit the -f, then the forked process gets an EACCES error trying to do ibv_query_device(). Thoughts?

Thanks,

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw
Quoted 4 lines · Steve Wise

The single case I see failing so far is trying to preload my library with netserver. If I run netserver with '-D -f' which avoids fork() calls, it works fine. If I omit the -f, then the forked process gets an EACCES error trying to do ibv_query_device(). Thoughts?

Hum, so I expect the patch to break things. Specifically if you change the process credentials after opening verbs. However just forking() shouldn't cause a problem. Can you confirm there is nothing that changes uid,gid, etc during the fork?

Unfortunately it was designed by Linus and Andy to work this way, so we cannot weaken the check.

I don't know how your preload library works, but after fork basically everything related to verbs is garbage - so it should be safe to close/reopen then uverbs fd which will avoid the problem. It is only because the FD was passed across a credential change that you got EACCESS.

The way forward to restore the lost functionality is to rework the entire uAPI to use ioctl, which is what Matan/Sean/etc have been doing. But that work seems to be going very slowly, I hoped we'd have something by now.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20160930175036.GA1867-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

author
#3 lore raw
Quoted 3 lines · Steve Wise
Quoted 2 lines

The single case I see failing so far is trying to preload my library

with

Quoted 1 line
Quoted 1 line

netserver. If I run netserver with '-D -f' which avoids fork() calls,

it

Quoted 1 line
Quoted 1 line

works fine. If I omit the -f, then the forked process gets an EACCES

error

Quoted 7 lines
Quoted 1 line

trying to do ibv_query_device(). Thoughts?

Hum, so I expect the patch to break things. Specifically if you change the process credentials after opening verbs. However just forking() shouldn't cause a problem. Can you confirm there is nothing that changes uid,gid, etc during the fork?

I'll debug this further, but this is running as root, so that sort of stuff shouldn't be changing.

Quoted 3 lines

Unfortunately it was designed by Linus and Andy to work this way, so we cannot weaken the check.

Sure.

Quoted 6 lines

I don't know how your preload library works, but after fork basically everything related to verbs is garbage - so it should be safe to close/reopen then uverbs fd which will avoid the problem. It is only because the FD was passed across a credential change that you got EACCESS.

Fork() is actually not supported by this preload library, but we were getting by, I guess, with netserver because it fork()s before it allocates the UDP socket that then gets accelerated by the preload lib. However, I'm thinking that librdacm is getting invoked and is opening contexts to all the verbs devices before the fork() and thus after it, this new "safe" check it biting me.

Quoted 3 lines

The way forward to restore the lost functionality is to rework the entire uAPI to use ioctl, which is what Matan/Sean/etc have been doing.

Right, I'm ok with waiting for this (and helping as I can, but I've been very quiet to date :) ).

Thanks Jason,

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 7 lines · Steve Wise
Quoted 4 lines

Hum, so I expect the patch to break things. Specifically if you change the process credentials after opening verbs. However just forking() shouldn't cause a problem. Can you confirm there is nothing that changes uid,gid, etc during the fork?

I'll debug this further, but this is running as root, so that sort of stuff shouldn't be changing.

Based on some emails I've got I think there is a 'thing' with credentials in Linux where something copies the creds struct but (perhaps?) does not change it.

Linus designed the test to be a pointer compare, so if anything does a copy-but-no-modify then it will break the test unnecessarily. If I understand everything properly :|

If you can add any more information I'm sure people will appreciate it.

Quoted 6 lines

Fork() is actually not supported by this preload library, but we were getting by, I guess, with netserver because it fork()s before it allocates the UDP socket that then gets accelerated by the preload lib. However, I'm thinking that librdacm is getting invoked and is opening contexts to all the verbs devices before the fork() and thus after it, this new "safe" check it biting me.

Hmm, sounds plausible. Same deal with rdmacm, it has to close/open the verbs device. Defering doing anything until first needed would also be a reasonable strategy.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20160930181705.GB1867-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

author
#6 lore raw
Quoted 7 lines · Steve Wise
Quoted 6 lines
Quoted 4 lines

Hum, so I expect the patch to break things. Specifically if you change the process credentials after opening verbs. However just forking() shouldn't cause a problem. Can you confirm there is nothing that changes uid,gid, etc during the fork?

I'll debug this further, but this is running as root, so that sort of

stuff

Quoted 6 lines
Quoted 1 line

shouldn't be changing.

Based on some emails I've got I think there is a 'thing' with credentials in Linux where something copies the creds struct but (perhaps?) does not change it.

Quoted 9 lines

Linus designed the test to be a pointer compare, so if anything does a copy-but-no-modify then it will break the test unnecessarily. If I understand everything properly :|

If you can add any more information I'm sure people will appreciate it.

Quoted 2 lines

Fork() is actually not supported by this preload library, but we were getting by, I guess, with netserver because it fork()s before it

allocates

Quoted 1 line
Quoted 1 line

the UDP socket that then gets accelerated by the preload lib. However,

I'm

Quoted 1 line
Quoted 1 line

thinking that librdacm is getting invoked and is opening contexts to all

the

Quoted 1 line
Quoted 1 line

verbs devices before the fork() and thus after it, this new "safe" check

it

Quoted 5 lines
Quoted 1 line

biting me.

Hmm, sounds plausible. Same deal with rdmacm, it has to close/open the verbs device. Defering doing anything until first needed would also be a reasonable strategy.

I think it does. But perhaps my offload lib is actually the culprit. It goes and walks all the interfaces and builds a database of available devices that can support this user mode UDP stack.

I'll post more as I identify the root cause.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html