Kernel tinification: shrinking the kernel and avoiding size regressions
Over time, the Linux kernel has grown far more featureful, but it has also grown significantly larger, even with all the optional features turned off. For the last several years, I've been working to make the kernel smaller, and mentoring/coordinating projects to do the same, to enable ridiculously small embedded applications and other fun uses. I'd like to discuss that work at Kernel Summit, get size regressions on the radar of kernel developers and subsystem maintainers, and solicit discussion on future possibilities to shrink the kernel further.
Topics:
- An overview of why the kernel's size still matters today ("but don't we all have tons of memory and storage?")
- Tiny in RAM versus tiny on storage.
- How much the kernel has grown over time.
- How size regressions happen and how to avoid them
- Size measurement, bloat-o-meter, allnoconfig, and other tools
- Compression and the decompression stub
- Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernel
- Discussion on proposed ways to make the kernel tiny, how much they might save, how much work they'd require, and how to implement them with minimal impact to the un-shrunken common case.
After the session, I'll prepare and maintain a detailed summary of the proposed ideas, ordered by how much space they'd save versus how much work they'd be. I plan to maintain that list on an ongoing basis, to coordinate tinification projects for ongoing work by people working on embedded applications, and for the benefit of mentorship projects such as OPW and SoC.
The set of people on the CC list would be good to have at the discussion. I marked this as "CORE TOPIC", but I'd also be happy to discuss it on the dual-track day if that's preferable.
(I'm currenly on the auto-generated nomination list.)
- Josh Triplett
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernel
Something that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
I first started thinking about this at LSF/MM where the subject of sys_remap_file_pages came up. "What even uses this?" "hardly anything". But for all the users that don't need it, there's this syscall always built in that does horrible things with VM internals. It's fortunate that there hasn't been anything particularly awful beyond simple DoS bugs in r_f_p.
Distribution kernels are in the sad position of having to always enable this stuff, but at least for people building their own kernels, or kernels for appliances, we could make their lives a little better by not even building this stuff in.
I had a patch to make this particular syscall a cond_syscall, but then XFS ate my homework and I haven't had chance to revisit this. So, my questions are:
- are there other obvious syscalls we could make optional without userspace freaking out when they suddenly start getting ENOSYS ?
- how much configurability here is too much ?
r_f_p was an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..
thoughts?
Dave
Quoted 11 lines · Dave Jones
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernelSomething that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
My first reaction is reducing the attack surface sounds a reasonable idea. My second reaction is that the plural in options makes me want to run for the hills. Having a sea of options for enabling and disabling syscalls gives us the potential for having a set of kernels all with a slightly differing ABI as people choose what to enable and disable.
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel CONFIG_SECURE_ROUTER which disables anything a secure router wouldn't need.
For the distros we could have an ordinary and a reduced attack surface kernel CONFIG_REDUCED_ATTACK_SURFACE.
The thing I really want to avoid is binaries compiled for one distro not running on another because of syscall differences.
Quoted 12 lines
I first started thinking about this at LSF/MM where the subject of
sys_remap_file_pagescame up. "What even uses this?" "hardly anything". But for all the users that don't need it, there's this syscall always built in that does horrible things with VM internals. It's fortunate that there hasn't been anything particularly awful beyond simple DoS bugs inr_f_p.Distribution kernels are in the sad position of having to always enable this stuff, but at least for people building their own kernels, or kernels for appliances, we could make their lives a little better by not even building this stuff in.
Quoted 6 lines
I had a patch to make this particular syscall a
cond_syscall, but then XFS ate my homework and I haven't had chance to revisit this. So, my questions are:
- are there other obvious syscalls we could make optional without userspace freaking out when they suddenly start getting ENOSYS ?
- how much configurability here is too much ?
I covered this above.
Quoted 3 lines
r_f_pwas an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..
Speaking with my Checkpoint/Restore/process migration container hat on, we need kcmp. It was designed with security in mind (originally we'd exposed kernel virtual addresses). Perhaps some of this hardening should be focussed more sharply on what is this syscall trying to do and could it achieve its aim in a more secure way.
James
Quoted 10 lines · James Bottomley
Quoted 4 lines
Something that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
My first reaction is reducing the attack surface sounds a reasonable idea. My second reaction is that the plural in options makes me want to run for the hills. Having a sea of options for enabling and disabling syscalls gives us the potential for having a set of kernels all with a slightly differing ABI as people choose what to enable and disable.
Absolutely. Deciding where the line is between 'core' syscalls, and optional ones is the tricky part.
Quoted 4 lines
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel
CONFIG_SECURE_ROUTERwhich disables anything a secure router wouldn't need.
That might be one option. And have these 'profiles' do select WANT_SYS_WHATEVER to reduce #ifdef complexity in the code..
Quoted 2 lines
For the distros we could have an ordinary and a reduced attack surface kernel
CONFIG_REDUCED_ATTACK_SURFACE.
Speaking from experience, when you have two flavors of something, over time things tend to creep. After a few "I want the reduced kernel but with xyz" requests, we're back to where we started, but I can certainly see people who do specialized kernels like tails wanting something like this.
Quoted 2 lines
The thing I really want to avoid is binaries compiled for one distro not running on another because of syscall differences.
This is one reason I think at least for general purpose distributions, this isn't really an option. Red Hat, SuSE etc _have_ to ship remap_file_pages because some tiny percentage of their userbase wants it.
Something else that might be worth thinking about would be a runtime method to disable syscalls. That might actually be more useful in the general case, but less so for the "I want a smaller build" crowd.
Quoted 7 lines
Quoted 3 lines
r_f_pwas an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..Speaking with my Checkpoint/Restore/process migration container hat on, we need kcmp. It was designed with security in mind (originally we'd exposed kernel virtual addresses).
Right, but as not everyone uses/needs checkpointing, it's an optional thing. I'm not proposing taking anything away for general purpose kernels here. kcmp is an example of the sort of thing I'd like to see more of.
Dave
Quoted 32 lines · Dave Jones
Quoted 10 lines · James Bottomley
Quoted 4 lines
Something that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
My first reaction is reducing the attack surface sounds a reasonable idea. My second reaction is that the plural in options makes me want to run for the hills. Having a sea of options for enabling and disabling syscalls gives us the potential for having a set of kernels all with a slightly differing ABI as people choose what to enable and disable.
Absolutely. Deciding where the line is between 'core' syscalls, and optional ones is the tricky part.
Quoted 4 lines
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel
CONFIG_SECURE_ROUTERwhich disables anything a secure router wouldn't need.That might be one option. And have these 'profiles' do select
WANT_SYS_WHATEVERto reduce #ifdef complexity in the code..Quoted 2 lines
For the distros we could have an ordinary and a reduced attack surface kernel
CONFIG_REDUCED_ATTACK_SURFACE.Speaking from experience, when you have two flavors of something, over time things tend to creep. After a few "I want the reduced kernel but with xyz" requests, we're back to where we started, but I can certainly see people who do specialized kernels like tails wanting something like this.
Quoted 6 lines
Quoted 2 lines
The thing I really want to avoid is binaries compiled for one distro not running on another because of syscall differences.
This is one reason I think at least for general purpose distributions, this isn't really an option. Red Hat, SuSE etc _have_ to ship
remap_file_pagesbecause some tiny percentage of their userbase wants it.
To mitigate that some, new syscalls could be added with CONFIG wrappers that default to disabled. The userbases can't use something that isn't explicitly turned on, and people would likely need to request those syscalls. It would give the distros at least a measure of how frequently that new syscall would be used, and in what situations.
josh
Quoted 8 lines · Josh Boyer
To mitigate that some, new syscalls could be added with CONFIG wrappers that default to disabled. The userbases can't use something that isn't explicitly turned on, and people would likely need to request those syscalls. It would give the distros at least a measure of how frequently that new syscall would be used, and in what situations.
In practice that is equivalent to not having the syscall at all.
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines · Josh Boyer
To mitigate that some, new syscalls could be added with CONFIG wrappers that default to disabled. The userbases can't use something that isn't explicitly turned on, and people would likely need to request those syscalls. It would give the distros at least a measure of how frequently that new syscall would be used, and in what situations.
In practice that is equivalent to not having the syscall at all.
Possibly. Typical case is that end users won't have those syscalls enabled and they won't care because nothing is using them. In the event that someone introduces something into the distro that does use it, you'd enable it, etc. That doesn't help the "one binary doesn't work on multiple distros" problem though, I guess.
My concern with CONFIG_ROUTER and other target profile variants is that it seems like an attempt at a system-wide seccomp of sorts, only via Kconfig options. You could accomplish similar things with SELinux or other security modules, so why would we go through the hassle of bugging people about syscall configs? I'm also skeptical that a general purpose distro would actually use anything but the broadest profile, but I'm not saying they wouldn't be useful.
josh
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines · Josh Boyer
To mitigate that some, new syscalls could be added with CONFIG wrappers that default to disabled. The userbases can't use something that isn't explicitly turned on, and people would likely need to request those syscalls. It would give the distros at least a measure of how frequently that new syscall would be used, and in what situations.
In practice that is equivalent to not having the syscall at all.
Amen!
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
There's been a huge focus on system calls in this discussion, and I suspect this is a bit of a red herring. Taking a look at "git log arch/x86/syscalls/syscall_64.tbl" --- since all the world's is no longer a Vax, but rather an x86_64 :-P --- there really hasn't been that many new system calls lately. Yes, we recently added renameat(2), but the next addition was half a year earlier, when the new schedular parameters syscalls went in.
There's much more in the way of kernel functionality and complexity which isn't really syscall related --- for example, all of the control group stuff, and security hair caused by things like user namespaces, and new fallocate(2) modes --- we've added PUNCH_HOLE, COLLAPSE_RANGE, and ZERO_RANGE, and there are threats to add INSERT_RANGE in the next release or two.
And if you look at things like renameat(2), the actual code savings by removing renameat(2) is pretty small, and IMHO, not worth the complexity and uncertainty that it would represent to application programmers of "does this system call exist or doesn't it".
In contrast, if you want to take at the bloat and complexity added by the pluggable security LSM's, control groups, and name spaces, the comparison isn't even close. Furthermore, given that low level progams programs like systemd have grown to require control groups, it's not like you can even realistically strip it from potentially even many embedded kernels, since there seems to be a movement to have systemd infect even smaller embedded applications.
Anyone want to lay odds on when systemd will start using various namespaces for its own purposes? :-)
- Ted
Quoted 8 lines · Theodore Ts'o
Furthermore, given that low level progams programs like systemd have grown to require control groups, it's not like you can even realistically strip it from potentially even many embedded kernels, since there seems to be a movement to have systemd infect even smaller embedded applications.
Anyone want to lay odds on when systemd will start using various namespaces for its own purposes? :-)
This is actually interesting topic on its own (but I am not completely what kernel maintainers could be realistically doing about it) -- userspace _strictly_ depending on _optional_ kernel features.
systemd and cgroups is a good example, but not the only one, I think. A few weeks ago I was struggling to boot a systemd-based userspace, to find out that fanotify needs to be turned on.
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 5 lines · Theodore Ts'o
There's been a huge focus on system calls in this discussion, and I suspect this is a bit of a red herring. Taking a look at "git log
arch/x86/syscalls/syscall_64.tbl" --- since all the world's is no longer a Vax, but rather anx86_64:-P --- there really hasn't been that many new system calls lately.
I may have a vested interest in syscalls :)
The rate we're adding them has slowed down, but the rate at which we're finding bugs exposed through them has accelerated enormously over the last few years.
To use just one example, on certain systems I'd love to be able to just turn off sys_perf_event_open given what a trainwreck of vulnerabilities it's been over the last few years [comedy: it is actually a config option, but x86 'selects' it, so you'll have it and you'll like it].
Thankfully at least the scarier parts of it are now hidden behind the paranoid sysctl.
Quoted 4 lines
And if you look at things like renameat(2), the actual code savings by removing renameat(2) is pretty small, and IMHO, not worth the complexity and uncertainty that it would represent to application programmers of "does this system call exist or doesn't it".
I think we've got two categories here.
"variant" syscalls like renameat, which just offers enhancements over an existing syscall. Stuff that things like glibc tend to care about. This stuff is usually pretty boring, and not even worth considering for potentially disabling imo.
And then we have "enable boatload of code" syscalls that are typically used by a few standalone apps/features. kexec, checkpointing, whatever db it was that cares about remap_file_pages, mempolicy, etc. etc.
It's this "not used by every user" code that tends to scare me, because it's written with 1-2 well behaved bits of userspace in mind, which usually means "has so many unchecked corner cases it's not even funny"
Ok, maybe there is also a grey area in the middle, which I guess depends on what your userspace is going to do, (things like vmsplice and friends), but I lean towards just classing them in the 2nd category too.
Quoted 7 lines
In contrast, if you want to take at the bloat and complexity added by the pluggable security LSM's, control groups, and name spaces, the comparison isn't even close. Furthermore, given that low level progams programs like systemd have grown to require control groups, it's not like you can even realistically strip it from potentially even many embedded kernels, since there seems to be a movement to have systemd infect even smaller embedded applications.
Yeah, we've reached a point of no return with things like cgroups now.
Quoted 2 lines
Anyone want to lay odds on when systemd will start using various namespaces for its own purposes? :-)
I thought it already was tbh.
Dave
Quoted 5 lines · On Fri, 2 May 2014 15:49:35 -0400 Dave Jones <[email protected]> wrote:
I may have a vested interest in syscalls :)
The rate we're adding them has slowed down, but the rate at which we're finding bugs exposed through them has accelerated enormously over the last few years.
Well, besides perf, I'm sure the acceleration of finding bugs wasn't due to system calls adding more of them, but because we now have much better ways of finding bugs (yes this is a kudos to your trinity).
-- Steve
Quoted 2 lines · Dave Jones
I may have a vested interest in syscalls :)
Well, yes, fair enough. :-)
Quoted 3 lines
And then we have "enable boatload of code" syscalls that are typically used by a few standalone apps/features. kexec, checkpointing, whatever db it was that cares about
remap_file_pages, mempolicy, etc. etc.
Sure, that's fair enough. I would just use a somewhat broader definition of "boatload of code", since not all of these boatloads are accessed via system calls. Some are accessed via new pseudo-file systems, some via new ioctls, some via new fallocate code points, etc.
Quoted 3 lines
It's this "not used by every user" code that tends to scare me, because it's written with 1-2 well behaved bits of userspace in mind, which usually means "has so many unchecked corner cases it's not even funny"
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid. It's the latter which is much more problematic from a security perspective --- as well from a "it's much harder to change the ABI without breaking large numbers of userspace programms". (Where as if it's only being used by 1-2 well behaved bits of privileged userspace, it's actually a tiny bit easier to evolve the ABI.)
So perhaps what that means it that _these_ are the features which require the most amount of paranoia and testing before we let them into the mainline kernel in the first place. Otherwise, once they get in, there's always a chance that systemd or some other piece of userspace will strict strictly requiring said optional feature, and it doesn't matter whether we put in a CONFIG option to disable the feature --- we'll never be able to do it.
- Ted
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Quoted 7 lines
So perhaps what that means it that _these_ are the features which require the most amount of paranoia and testing before we let them into the mainline kernel in the first place. Otherwise, once they get in, there's always a chance that systemd or some other piece of userspace will strict strictly requiring said optional feature, and it doesn't matter whether we put in a CONFIG option to disable the feature --- we'll never be able to do it.
This is starting to tread into the other thread about userspace mandating 'optional' facilities, but is that even a problem, given the proliferation of init's (taking the systemd example). Yes, systemd "won" by now being the default in all the general purpose distributions, but with my upstream hat on, I think we still care about embedded systems etc that don't need anywhere near the functionality that systemd provides.
Dave
[*] oh god, ioctl.
Quoted 8 lines · Dave Jones
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
josh
Quoted 4 lines · Josh Boyer
Quoted 4 lines
Quoted 3 lines
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
Quoted 8 lines
Quoted 2 lines
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
I believe the biggest remaining thing on the plate is basically just kexec/kdump ... is there anything else comparably major?
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 9 lines · Jiri Kosina
Quoted 7 lines · Josh Boyer
Quoted 6 lines
Quoted 3 lines
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Quoted 8 lines
Quoted 5 lines
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
I believe the biggest remaining thing on the plate is basically just kexec/kdump ... is there anything else comparably major?
Alan seems to entirely disagree with the approach we've been taking for the past couple of years. I would have to go back and re-read things, but he seems to favor some rework of capabilities into a matrix somehow. I have no idea if he's been working on this or if it's feasible, or even if anyone else favors that approach.
The dissent seems to have been enough to get the trusted_kernel patches not pulled into the security tree, so we probably should discuss it at least a little.
josh
Quoted 7 lines · Josh Boyer
Quoted 6 lines · Dave Jones
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
Quoted 8 lines
Quoted 2 lines
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
Would you believe we're just discussing with the distros how we might re-engineer the Linux secure boot process. Unfortunately the details depend on a UEFI forum proposal that are UEFI confidential at this time, but you can probably pick them up from Peter Jones, since you're a Red Hat employee. One of the side effects of this, if it happens, will be to separate Linux secure boot policy from Microsoft's binary signing requirements which might take some of the heat out of the arguments about which parts of the patch are to please microsoft and refocus the debate towards how we make better use of secure boot. I'll try and ensure that either the proposals are public by KS or that we have permission to share the details.
James
Quoted 11 lines · James Bottomley
Quoted 10 lines · Josh Boyer
Quoted 8 lines · Dave Jones
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Quoted 8 lines
Quoted 5 lines
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
Would you believe we're just discussing with the distros how we might re-engineer the Linux secure boot process. Unfortunately the details
I would believe it.
Quoted 3 lines
depend on a UEFI forum proposal that are UEFI confidential at this time, but you can probably pick them up from Peter Jones, since you're a Red Hat employee. One of the side effects of this, if it happens, will be
OK.
Quoted 6 lines
to separate Linux secure boot policy from Microsoft's binary signing requirements which might take some of the heat out of the arguments about which parts of the patch are to please microsoft and refocus the debate towards how we make better use of secure boot. I'll try and ensure that either the proposals are public by KS or that we have permission to share the details.
The objectionable parts having to do with signing aren't even in the patchset Matthew has posted. That's the initial set he tried to get pulled in and failed. If the proposal drastically changes that approach I'd be surprised (maybe pleasantly).
josh
Quoted 37 lines · Josh Boyer
Quoted 11 lines · James Bottomley
Quoted 10 lines · Josh Boyer
Quoted 8 lines · Dave Jones
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Quoted 8 lines
Quoted 5 lines
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
Would you believe we're just discussing with the distros how we might re-engineer the Linux secure boot process. Unfortunately the details
I would believe it.
Quoted 3 lines
depend on a UEFI forum proposal that are UEFI confidential at this time, but you can probably pick them up from Peter Jones, since you're a Red Hat employee. One of the side effects of this, if it happens, will be
OK.
Quoted 6 lines
to separate Linux secure boot policy from Microsoft's binary signing requirements which might take some of the heat out of the arguments about which parts of the patch are to please microsoft and refocus the debate towards how we make better use of secure boot. I'll try and ensure that either the proposals are public by KS or that we have permission to share the details.
Quoted 4 lines
The objectionable parts having to do with signing aren't even in the patchset Matthew has posted. That's the initial set he tried to get pulled in and failed. If the proposal drastically changes that approach I'd be surprised (maybe pleasantly).
FWIW, I really don't like the approach where we say that the kernel must be inviolate but that user code can do whatever it likes as long as the kernel isn't compromised. This may be needed to comply with current MS/UEFI policy, but I think it largely misses the point wrt actual security.
If the policy can change, then that might be a huge win.
--Andy
Quoted 43 lines · Andy Lutomirski
Quoted 37 lines · Josh Boyer
Quoted 11 lines · James Bottomley
Quoted 10 lines · Josh Boyer
Quoted 8 lines · Dave Jones
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Quoted 8 lines
Quoted 5 lines
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
Would you believe we're just discussing with the distros how we might re-engineer the Linux secure boot process. Unfortunately the details
I would believe it.
Quoted 3 lines
depend on a UEFI forum proposal that are UEFI confidential at this time, but you can probably pick them up from Peter Jones, since you're a Red Hat employee. One of the side effects of this, if it happens, will be
OK.
Quoted 6 lines
to separate Linux secure boot policy from Microsoft's binary signing requirements which might take some of the heat out of the arguments about which parts of the patch are to please microsoft and refocus the debate towards how we make better use of secure boot. I'll try and ensure that either the proposals are public by KS or that we have permission to share the details.
Quoted 4 lines
The objectionable parts having to do with signing aren't even in the patchset Matthew has posted. That's the initial set he tried to get pulled in and failed. If the proposal drastically changes that approach I'd be surprised (maybe pleasantly).
Quoted 7 lines
FWIW, I really don't like the approach where we say that the kernel must be inviolate but that user code can do whatever it likes as long as the kernel isn't compromised. This may be needed to comply with current MS/UEFI policy, but I think it largely misses the point wrt actual security.
If the policy can change, then that might be a huge win.
We shouldn't give up on securing userspace, either; protecting the kernel is necessary but not sufficient. But I do think it's worthwhile to enforce "root != kernel", quite apart from any "secure boot" requirements. That's what Matthew's patches primarily serve to do: make root not kernel-equivalent.
- Josh Triplett
Quoted 50 lines · On Tue, May 6, 2014 at 10:18 AM, <[email protected]> wrote:
Quoted 49 lines · Andy Lutomirski
Quoted 37 lines · Josh Boyer
Quoted 11 lines · James Bottomley
Quoted 10 lines · Josh Boyer
Quoted 8 lines · Dave Jones
Quoted 3 lines · Theodore Ts'o
And I think we can also further break this down into the classes of code which require root privs (i.e., like kexec), and those which can be used by any userid.
In the brave new world of secure boot, we kind of have to care about even the root cases now too [*], but I agree in the general case.
Quoted 8 lines
Quoted 5 lines
Speaking of that... is it worth my time to propose a "What to do about the secure_modules/trusted_kernel/whatever patch set that distros are carrying to support Secure Boot? I thought we had agreement and a path forward at LPC last year, but things seem to have gotten derailed again.
Would you believe we're just discussing with the distros how we might re-engineer the Linux secure boot process. Unfortunately the details
I would believe it.
Quoted 3 lines
depend on a UEFI forum proposal that are UEFI confidential at this time, but you can probably pick them up from Peter Jones, since you're a Red Hat employee. One of the side effects of this, if it happens, will be
OK.
Quoted 6 lines
to separate Linux secure boot policy from Microsoft's binary signing requirements which might take some of the heat out of the arguments about which parts of the patch are to please microsoft and refocus the debate towards how we make better use of secure boot. I'll try and ensure that either the proposals are public by KS or that we have permission to share the details.
Quoted 4 lines
The objectionable parts having to do with signing aren't even in the patchset Matthew has posted. That's the initial set he tried to get pulled in and failed. If the proposal drastically changes that approach I'd be surprised (maybe pleasantly).
FWIW, I really don't like the approach where we say that the kernel must be inviolate but that user code can do whatever it likes as long as the kernel isn't compromised. This may be needed to comply with current MS/UEFI policy, but I think it largely misses the point wrt actual security.
Quoted 8 lines
Quoted 1 line
If the policy can change, then that might be a huge win.
We shouldn't give up on securing userspace, either; protecting the kernel is necessary but not sufficient. But I do think it's worthwhile to enforce "root != kernel", quite apart from any "secure boot" requirements. That's what Matthew's patches primarily serve to do: make root not kernel-equivalent.
I have two main objections to "root != kernel". The bigger is that I'd like to see the security argument for it so that people can think about whether it makes sense. The smaller is that "root != kernel" isn't necessarily well-defined.
For example, should root be able to write to the filesystem from which the kernel loads? Should root be able to kexec a new kernel, if that clears some key known to the current kernel in the process? Should root be able to start a KVM instance that passes essentially all hardware through? Should root be able to talk directly to the system's embedded controller? Should root be able to read all physical memory? How about reading just enough to learn the kernel's semi-secret randomized addresses? How about running perf without restrictions?
In the past, the actual security goal seems to have been "root shall not be able to do anything that would anger Microsoft and/or Verisign", which is far-enough removed from actual security that I don't want it anywhere near my system. But if I could have a reasonable policy that "root shall not be able to persistently compromise the machine", then I think this could be great.
Note that the latter goal does not actually require that root be unable to modify the running kernel.
--Andy
Quoted 23 lines · Andy Lutomirski
I have two main objections to "root != kernel". The bigger is that I'd like to see the security argument for it so that people can think about whether it makes sense. The smaller is that "root != kernel" isn't necessarily well-defined.
For example, should root be able to write to the filesystem from which the kernel loads? Should root be able to kexec a new kernel, if that clears some key known to the current kernel in the process? Should root be able to start a KVM instance that passes essentially all hardware through? Should root be able to talk directly to the system's embedded controller? Should root be able to read all physical memory? How about reading just enough to learn the kernel's semi-secret randomized addresses? How about running perf without restrictions?
In the past, the actual security goal seems to have been "root shall not be able to do anything that would anger Microsoft and/or Verisign", which is far-enough removed from actual security that I don't want it anywhere near my system. But if I could have a reasonable policy that "root shall not be able to persistently compromise the machine", then I think this could be great.
Quoted 3 lines
Note that the latter goal does not actually require that root be unable to modify the running kernel.
The first aspect of this is that the kernel needs to *be able to* lock out root from select functions. These things will be system configuration dependent.
Once you have the separation you can define exceptions.
-hpa
Quoted 24 lines · H. Peter Anvin
Quoted 23 lines · Andy Lutomirski
I have two main objections to "root != kernel". The bigger is that I'd like to see the security argument for it so that people can think about whether it makes sense. The smaller is that "root != kernel" isn't necessarily well-defined.
For example, should root be able to write to the filesystem from which the kernel loads? Should root be able to kexec a new kernel, if that clears some key known to the current kernel in the process? Should root be able to start a KVM instance that passes essentially all hardware through? Should root be able to talk directly to the system's embedded controller? Should root be able to read all physical memory? How about reading just enough to learn the kernel's semi-secret randomized addresses? How about running perf without restrictions?
In the past, the actual security goal seems to have been "root shall not be able to do anything that would anger Microsoft and/or Verisign", which is far-enough removed from actual security that I don't want it anywhere near my system. But if I could have a reasonable policy that "root shall not be able to persistently compromise the machine", then I think this could be great.
Quoted 8 lines
Quoted 3 lines
Note that the latter goal does not actually require that root be unable to modify the running kernel.
The first aspect of this is that the kernel needs to *be able to* lock out root from select functions. These things will be system configuration dependent.
I'm still unconvinced. For Chrome OS-style security, I think that root just needs to be prevented from doing anything that will interfere with the verified boot process the next time the machine boots. The kernel doesn't need any particular security feature for
this: the kernel can't change the verified boot keys either. If an attacker controls root on a Chromebook, the attacker has already won, at least until the next reboot.
If the idea is to have a verified boot without any hardware or firmware support, then, yes, the kernel needs to enforce that the verification path can't be tampered with. But I think we're talking about Secure Boot here, and on a correct Secure Boot implementation*, the worst that the kernel can do is to prevent the box from booting next time.
The best arguments I've heard so far for why the kernel needs to try to protect itself against root are:
1. MS/Verisign demand it.
2. It's annoying to fool a user into thinking that they just booted Some Other OS when they're really running Linux without kernel help.
NB: no one has claimed that it's impossible AFAIK, just that it's annoyingly complicated.
I like neither of these arguments. #1 is politics, not security, and #2 seems like security by annoying the attacker.
To be clear, I don't object on principle to making it possible for the kernel to defend itself against root. But it's hard, doing it right will require a lot of care, and I don't think it's worth doing unless there's a good reason. If there's a good reason that I don't know about, please tell me!
* The recent MITRE paper suggests that very few of these exist. That's a separate issue.
--Andy
Quoted 5 lines · Andy Lutomirski
The best arguments I've heard so far for why the kernel needs to try to protect itself against root are:
1. MS/Verisign demand it.
Quoted 7 lines
2. It's annoying to fool a user into thinking that they just booted Some Other OS when they're really running Linux without kernel help.
NB: no one has claimed that it's impossible AFAIK, just that it's annoyingly complicated.I like neither of these arguments. #1 is politics, not security, and #2 seems like security by annoying the attacker.
#1 is useful if you care about supporting users booting Linux on modern systems without changing BIOS configuration.
As for #2, I agree that it's just "annoying the attacker", and I don't want to quibble over the value of that in this particular case, but keep in mind that a *lot* of security is "annoying the attacker"; you can rather precisely quantify how secure a system is by how much it costs to purchase exploited systems or similar. (See "An Agenda for Empirical Cyber Crime Research", USENIX ATC 2011.) And in very much the same spirit as "I don't have to run faster than the bear", a lot of security (against broad-scale exploits rather than targeted threats) is about making it more painful to exploit a system than to do a social-engineering attack or a physical security breach.
- Josh Triplett
Quoted 49 lines · Andy Lutomirski
Quoted 24 lines · H. Peter Anvin
Quoted 23 lines · Andy Lutomirski
I have two main objections to "root != kernel". The bigger is that I'd like to see the security argument for it so that people can think about whether it makes sense. The smaller is that "root != kernel" isn't necessarily well-defined.
For example, should root be able to write to the filesystem from which the kernel loads? Should root be able to kexec a new kernel, if that clears some key known to the current kernel in the process? Should root be able to start a KVM instance that passes essentially all hardware through? Should root be able to talk directly to the system's embedded controller? Should root be able to read all physical memory? How about reading just enough to learn the kernel's semi-secret randomized addresses? How about running perf without restrictions?
In the past, the actual security goal seems to have been "root shall not be able to do anything that would anger Microsoft and/or Verisign", which is far-enough removed from actual security that I don't want it anywhere near my system. But if I could have a reasonable policy that "root shall not be able to persistently compromise the machine", then I think this could be great.
Quoted 8 lines
Quoted 3 lines
Note that the latter goal does not actually require that root be unable to modify the running kernel.
The first aspect of this is that the kernel needs to *be able to* lock out root from select functions. These things will be system configuration dependent.
I'm still unconvinced. For Chrome OS-style security, I think that root just needs to be prevented from doing anything that will interfere with the verified boot process the next time the machine boots. The kernel doesn't need any particular security feature for
this: the kernel can't change the verified boot keys either. If an attacker controls root on a Chromebook, the attacker has already won, at least until the next reboot.If the idea is to have a verified boot without any hardware or firmware support, then, yes, the kernel needs to enforce that the verification path can't be tampered with. But I think we're talking about Secure Boot here, and on a correct Secure Boot implementation*, the worst that the kernel can do is to prevent the box from booting next time.
Quoted 4 lines
The best arguments I've heard so far for why the kernel needs to try to protect itself against root are:
1. MS/Verisign demand it.
I think we should stop focussing on this case because firstly it's second guessing what we actually have to do and secondly there might be a solution on the horizon for Secure Boot which removes the issue. As I said, I'll try to have non uefi confidential details by the time KS rolls around.
Quoted 5 lines
2. It's annoying to fool a user into thinking that they just booted Some Other OS when they're really running Linux without kernel help.
NB: no one has claimed that it's impossible AFAIK, just that it's annoyingly complicated.
Quoted 8 lines
I like neither of these arguments. #1 is politics, not security, and #2 seems like security by annoying the attacker.
To be clear, I don't object on principle to making it possible for the kernel to defend itself against root. But it's hard, doing it right will require a lot of care, and I don't think it's worth doing unless there's a good reason. If there's a good reason that I don't know about, please tell me!
I can be agnostic on this. What I want is something that gives my internet attached server more security ... in particular, if someone breaks in I want them to be able to do as little damage as possible. The reason I'm agnostic is that I have had several break in's over the past decade I've been running my own colo hosted system. None actually managed to escalate to root (so any root security wouldn't have helped). The problem for me was that they managed to do enough damage without being root (one even managed to get apache to send out masses of spam leaving me with a huge cleanup operation ... lesson learned, I now have an iptables OUTPUT rule rejecting any non postfix traffic to port 25 and alerting me to the attempt).
The point is I like the theoretical idea of protecting my system from someone who manages to escalate to root and hey, if it can be done easily, I'll take it, but that's not usually my biggest problem when I consider security.
James
Quoted 87 lines · James Bottomley
Quoted 49 lines · Andy Lutomirski
Quoted 24 lines · H. Peter Anvin
Quoted 23 lines · Andy Lutomirski
I have two main objections to "root != kernel". The bigger is that I'd like to see the security argument for it so that people can think about whether it makes sense. The smaller is that "root != kernel" isn't necessarily well-defined.
For example, should root be able to write to the filesystem from which the kernel loads? Should root be able to kexec a new kernel, if that clears some key known to the current kernel in the process? Should root be able to start a KVM instance that passes essentially all hardware through? Should root be able to talk directly to the system's embedded controller? Should root be able to read all physical memory? How about reading just enough to learn the kernel's semi-secret randomized addresses? How about running perf without restrictions?
In the past, the actual security goal seems to have been "root shall not be able to do anything that would anger Microsoft and/or Verisign", which is far-enough removed from actual security that I don't want it anywhere near my system. But if I could have a reasonable policy that "root shall not be able to persistently compromise the machine", then I think this could be great.
Quoted 8 lines
Quoted 3 lines
Note that the latter goal does not actually require that root be unable to modify the running kernel.
The first aspect of this is that the kernel needs to *be able to* lock out root from select functions. These things will be system configuration dependent.
I'm still unconvinced. For Chrome OS-style security, I think that root just needs to be prevented from doing anything that will interfere with the verified boot process the next time the machine boots. The kernel doesn't need any particular security feature for
this: the kernel can't change the verified boot keys either. If an attacker controls root on a Chromebook, the attacker has already won, at least until the next reboot.If the idea is to have a verified boot without any hardware or firmware support, then, yes, the kernel needs to enforce that the verification path can't be tampered with. But I think we're talking about Secure Boot here, and on a correct Secure Boot implementation*, the worst that the kernel can do is to prevent the box from booting next time.
Quoted 4 lines
The best arguments I've heard so far for why the kernel needs to try to protect itself against root are:
1. MS/Verisign demand it.
I think we should stop focussing on this case because firstly it's second guessing what we actually have to do and secondly there might be a solution on the horizon for Secure Boot which removes the issue. As I said, I'll try to have non uefi confidential details by the time KS rolls around.
Quoted 5 lines
2. It's annoying to fool a user into thinking that they just booted Some Other OS when they're really running Linux without kernel help.
NB: no one has claimed that it's impossible AFAIK, just that it's annoyingly complicated.Quoted 8 lines
I like neither of these arguments. #1 is politics, not security, and #2 seems like security by annoying the attacker.
To be clear, I don't object on principle to making it possible for the kernel to defend itself against root. But it's hard, doing it right will require a lot of care, and I don't think it's worth doing unless there's a good reason. If there's a good reason that I don't know about, please tell me!
I can be agnostic on this. What I want is something that gives my internet attached server more security ... in particular, if someone breaks in I want them to be able to do as little damage as possible. The reason I'm agnostic is that I have had several break in's over the past decade I've been running my own colo hosted system. None actually managed to escalate to root (so any root security wouldn't have helped). The problem for me was that they managed to do enough damage without being root (one even managed to get apache to send out masses of spam leaving me with a huge cleanup operation ... lesson learned, I now have an iptables OUTPUT rule rejecting any non postfix traffic to port 25 and alerting me to the attempt).
Quoted 4 lines
The point is I like the theoretical idea of protecting my system from someone who manages to escalate to root and hey, if it can be done easily, I'll take it, but that's not usually my biggest problem when I consider security.
We're almost at the point where it would be reasonable to shove basically every service on a system into a user namespace, in which case, barring bugs, you shouldn't be able to own the kernel. I wonder if this might do pretty much exactly what you want.
Essentially, you'd mount your filesystems, make a new userns, move all network devices into a new netns owned by that userns, unshare the mount namespace, and somehow get systemd or whatever other init program you're using to play along.
If you're just trying to protect the kernel, you can even map all uids straight through, including root.
The major annoyance is that it's damn near impossible to mount real filesystems (i.e. not tmpfs) from inside a userns. There could be a daemon outside that helps out.
This doesn't necessarily do anything sensible with device nodes, but that might not matter for servers.
--Andy
Quoted 4 lines · Andy Lutomirski
We're almost at the point where it would be reasonable to shove basically every service on a system into a user namespace, in which case, barring bugs, you shouldn't be able to own the kernel. I wonder if this might do pretty much exactly what you want.
We should absolutely do this, and it'll make a big difference. However, "barring bugs" is a pretty big bar; in practice, it's probably easier to get from user->kernel than to get from user->root, just because you can do the former from any process that can make system calls. We're not anywhere close to done with fixing system call vulnerabilities.
Quoted 4 lines
Essentially, you'd mount your filesystems, make a new userns, move all network devices into a new netns owned by that userns, unshare the mount namespace, and somehow get systemd or whatever other init program you're using to play along.
systemd makes it rather easy to configure a service for this kind of namespace isolation. You can, for instance, put services that don't need the network in a network namespace that only includes localhost. I suspect that far more services will take advantage of that than will attempt to configure an equivalent isolation setup manually.
- Josh Triplett
Apologies for not seeing this. Apparently this list is set up with nodupes as the default and I have mail rules trashing personal copies of mail I'm supposed to get from the list
Quoted 9 lines · Josh Boyer
Quoted 6 lines · James Bottomley
to separate Linux secure boot policy from Microsoft's binary signing requirements which might take some of the heat out of the arguments about which parts of the patch are to please microsoft and refocus the debate towards how we make better use of secure boot. I'll try and ensure that either the proposals are public by KS or that we have permission to share the details.
Quoted 4 lines
The objectionable parts having to do with signing aren't even in the patchset Matthew has posted. That's the initial set he tried to get pulled in and failed. If the proposal drastically changes that approach I'd be surprised (maybe pleasantly).
Some of the objections are rooted in the suspicions that what we do, we do to please Microsoft (or at least to get them not to blacklist our signatures) others are simply based on the idea that secure boot isn't, because Microsoft designed it wrongly, so we shouldn't call it secure.
Removing the Microsoft proxy allows us to have a more honest debate about how we want to make use of the capability. I'm not saying it produces immediate agreement because there's plenty of stuff we have disagreements over that aren't rooted in suspicions of ulterior motives, but at least we'll be disagreeing about real issues.
James
Quoted 3 lines · Dave Jones
This is starting to tread into the other thread about userspace mandating 'optional' facilities, but is that even a problem, given the proliferation of init's (taking the systemd example).
For the record, there is no "other thread" because as Jiri pointed out (and I agree) it's not at all clear there's anything we as kernel developers can do about this. Once some kind of system call interface is made available in mainline, it's not at all realistic to say that "it's only for this use case".
If someone thinks that it might be worthwhile to start such a thread and/or propose that that might be a worthy topic for the kernel summit, feel free to start such a thread --- but I'd suggest doing so after coming up with a proposed solution to how we could even influence what facilities various bits of userspace might want to mandate.
Cheers,
- Ted
Quoted 13 lines · Dave Jones
Quoted 5 lines · Theodore Ts'o
There's been a huge focus on system calls in this discussion, and I suspect this is a bit of a red herring. Taking a look at "git log
arch/x86/syscalls/syscall_64.tbl" --- since all the world's is no longer a Vax, but rather anx86_64:-P --- there really hasn't been that many new system calls lately.I may have a vested interest in syscalls :)
The rate we're adding them has slowed down, but the rate at which we're finding bugs exposed through them has accelerated enormously over the last few years.
Quoted 6 lines
To use just one example, on certain systems I'd love to be able to just turn off
sys_perf_event_opengiven what a trainwreck of vulnerabilities it's been over the last few years [comedy: it is actually a config option, but x86 'selects' it, so you'll have it and you'll like it].
Thankfully at least the scarier parts of it are now hidden behind the paranoid sysctl.
I have considered proposing perf_event_paranoid=3 to disable it completely for non-root.
Quoted 12 lines
Quoted 4 lines
And if you look at things like renameat(2), the actual code savings by removing renameat(2) is pretty small, and IMHO, not worth the complexity and uncertainty that it would represent to application programmers of "does this system call exist or doesn't it".
I think we've got two categories here.
"variant" syscalls like renameat, which just offers enhancements over an existing syscall. Stuff that things like glibc tend to care about. This stuff is usually pretty boring, and not even worth considering for potentially disabling imo.
Quoted 7 lines
And then we have "enable boatload of code" syscalls that are typically used by a few standalone apps/features. kexec, checkpointing, whatever db it was that cares about
remap_file_pages, mempolicy, etc. etc.It's this "not used by every user" code that tends to scare me, because it's written with 1-2 well behaved bits of userspace in mind, which usually means "has so many unchecked corner cases it's not even funny"
[...]
Since Michael often seems to be the one testing those corner cases while writing documentation, it seems like you're getting back to the old issue of whether lack of documentation should be a blocker for adding new system calls.
Ben.
Signature · 3 lines
Ben Hutchings
Lowery's Law:
If it jams, force it. If it breaks, it needed replacing anyway.Quoted 11 lines · Ben Hutchings
Quoted 6 lines · Dave Jones
To use just one example, on certain systems I'd love to be able to just turn off
sys_perf_event_opengiven what a trainwreck of vulnerabilities it's been over the last few years [comedy: it is actually a config option, but x86 'selects' it, so you'll have it and you'll like it].
Thankfully at least the scarier parts of it are now hidden behind the paranoid sysctl.I have considered proposing perf_event_paranoid=3 to disable it completely for non-root.
Doesn't seem too crazy an idea to me.
Quoted 9 lines
Quoted 3 lines
It's this "not used by every user" code that tends to scare me, because it's written with 1-2 well behaved bits of userspace in mind, which usually means "has so many unchecked corner cases it's not even funny"
[...]
Since Michael often seems to be the one testing those corner cases while writing documentation, it seems like you're getting back to the old issue of whether lack of documentation should be a blocker for adding new system calls.
That, and test cases.
Dave
Quoted 2 lines · Dave Jones
Quoted 1 line · Ben Hutchings
On Fri, 2014-05-02 at 15:49 -0400, Dave Jones wrote:
[...]
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
It's this "not used by every user" code that tends to scare me, because it's written with 1-2 well behaved bits of userspace in mind, which usually means "has so many unchecked corner cases it's not even funny"
[...]
Since Michael often seems to be the one testing those corner cases while writing documentation, it seems like you're getting back to the old issue of whether lack of documentation should be a blocker for adding new system calls.
That, and test cases.
Form my point of view, test cases and documentation go hand in glove (see my last mail).
Hmmm -- this almost starts to smell like a topic to revisit at Ksummit.
Thanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Quoted 8 lines · Ben Hutchings
Quoted 7 lines · Dave Jones
Quoted 5 lines · Theodore Ts'o
There's been a huge focus on system calls in this discussion, and I suspect this is a bit of a red herring. Taking a look at "git log
arch/x86/syscalls/syscall_64.tbl" --- since all the world's is no longer a Vax, but rather anx86_64:-P --- there really hasn't been that many new system calls lately.
Quoted 5 lines
Quoted 5 lines
I may have a vested interest in syscalls :)
The rate we're adding them has slowed down, but the rate at which we're finding bugs exposed through them has accelerated enormously over the last few years.
Yes. The APIs delivered to userspace continue to be infested with bugs and design infelicities, many of which go undetected for a long time.
Quoted 7 lines
Quoted 6 lines
To use just one example, on certain systems I'd love to be able to just turn off
sys_perf_event_opengiven what a trainwreck of vulnerabilities it's been over the last few years [comedy: it is actually a config option, but x86 'selects' it, so you'll have it and you'll like it].
Thankfully at least the scarier parts of it are now hidden behind the paranoid sysctl.
Quoted 22 lines
I have considered proposing perf_event_paranoid=3 to disable it completely for non-root.
Quoted 19 lines
Quoted 4 lines
And if you look at things like renameat(2), the actual code savings by removing renameat(2) is pretty small, and IMHO, not worth the complexity and uncertainty that it would represent to application programmers of "does this system call exist or doesn't it".
I think we've got two categories here.
"variant" syscalls like renameat, which just offers enhancements over an existing syscall. Stuff that things like glibc tend to care about. This stuff is usually pretty boring, and not even worth considering for potentially disabling imo.
And then we have "enable boatload of code" syscalls that are typically used by a few standalone apps/features. kexec, checkpointing, whatever db it was that cares about
remap_file_pages, mempolicy, etc. etc.It's this "not used by every user" code that tends to scare me, because it's written with 1-2 well behaved bits of userspace in mind, which usually means "has so many unchecked corner cases it's not even funny"
Well it's worse than that, I think. Those unchecked corner cases turn up even in code that is not protected by config options or privs. My example of the day: the timeout argument of recvmmsg() does nothing sensible--there was no (or minimal) testing, seems to have been minimal review of the feature, and of course there was no documentation of how the timeout feature should work beyond the statement that "recvmmsg now has a struct timespec timeout, that works in the same fashion as the ppoll one" (Newsflash: recvmmsg() and ppoll() are doing very different things, so describing one in terms of the other doesn't provide much insight.)
https://bugzilla.kernel.org/show_bug.cgi?id=75371
http://thread.gmane.org/gmane.linux.man/5677
Quoted 6 lines
[...]
Since Michael often seems to be the one testing those corner cases while writing documentation, it seems like you're getting back to the old issue of whether lack of documentation should be a blocker for adding new system calls.
I think there's really room for a lot more rigor here. There is way too much crap hitting the userspace API. I've long argued that (ggod) documentation is one of the best ways of finding bugs and design errors. I know, because that's the way I've discovered a lot of the problems. Of course, perhaps I am just an odd data point, but I recently got to help out in an experiment that reproduced the results.
Heinrich Schuchardt recently took it upon himself to document the fanotify API, which has been undocumented since its release in 2.6.37. (Heinrich's pages will probably be published in the next week or so, in the meantime the drafts are here:
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/tree/ )
In the course of writing the pages (and goaded by me at various points to "explain this detail" or "tell the reader what happens in this case"), Heinrich has uncovered (and documented) one or two design infelicities and a good crop of bugs (at least one of which has some security implications:
http://thread.gmane.org/gmane.linux.kernel/1686672/focus=1690201 )
So, Heinrich demonstrated what I've long known: show me a new kernel-user-space API and I can probably pretty quickly show you a bug. Writing good documentation goes a long way toward finding those bugs and design problems, and it really should be done well before an API is released, since, of course, some API problems can't be fixed later. And, it should be a collaborative effort involving not just the developer concerned but someone fairly distant from them who can look skeptically at the documentation.
Oh, and I didn't explicitly say it, but to me it's obvious:
good documentation necessarily implies good testing. And that's the thing that made Heinrich's work good: when he wrote in response to some of my goadings that the answers might take a while, because he'd need to write some tests, that was exactly what I hoped to hear.
tools like trinity do a great job of catching bizarre behaviors in APIs, but in the end some bugs (and design problems) are only going to be found when human beings sit down and think deeply about what is going on. (The timeout issue for recvmmsg() is a case in point. There's no fuzz testing for that sort of issue, and for that matter no specification of the expected behavior against which to test.)
Thanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Quoted 7 lines · Theodore Ts'o
There's been a huge focus on system calls in this discussion, and I suspect this is a bit of a red herring. Taking a look at "git log
arch/x86/syscalls/syscall_64.tbl" --- since all the world's is no longer a Vax, but rather anx86_64:-P --- there really hasn't been that many new system calls lately. Yes, we recently added renameat(2), but the next addition was half a year earlier, when the new schedular parameters syscalls went in.
A minor correction: that wasn't 6 months ago -- it was 3.14, released at the end of March, that added sched_getattr() and sched_setattr().
Quoted 6 lines
There's much more in the way of kernel functionality and complexity which isn't really syscall related --- for example, all of the control group stuff, and security hair caused by things like user namespaces, and new fallocate(2) modes --- we've added
PUNCH_HOLE,COLLAPSE_RANGE, andZERO_RANGE, and there are threats to addINSERT_RANGEin the next release or two.
Yes, that's a much bigger part of the growing surface. (Just by the bye, I try to track the growth of the surface at
http://man7.org/tlpi/api_changes/ . Corrections and additions are welcome. It's reasonably complete with respect to system calls, partially complete on /proc and socket options, and rather out to lunch on other pieces such as /sys and other pseudo filesystems
Cheers,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Quoted 3 lines · Dave Jones
Something else that might be worth thinking about would be a runtime method to disable syscalls. That might actually be more useful in the general case, but less so for the "I want a smaller build" crowd.
It would be useful for the smaller build case to have a way of auditing which syscalls are actually in use on a system so you can then go through and construct a minimal config.
Quoted 3 lines
It would be useful for the smaller build case to have a way of auditing which syscalls are actually in use on a system so you can then go through and construct a minimal config.
"strace -c" ?
-Tony
Quoted 5 lines · Tony
Quoted 3 lines
It would be useful for the smaller build case to have a way of auditing which syscalls are actually in use on a system so you can then go through and construct a minimal config.
"strace -c" ?
That works for specific processes but I don't immediately see a straightforward way to do it system wide (I guess a wrapper that straces init and children might do the trick but it's not particularly nice). Part of the trick for getting the general security win is to lower the barrier to entry.`
Quoted 6 lines · Mark Brown
Quoted 3 lines · Tony
Quoted 3 lines
It would be useful for the smaller build case to have a way of auditing which syscalls are actually in use on a system so you can then go through and construct a minimal config.
Quoted 7 lines
Quoted 1 line
"strace -c" ?
That works for specific processes but I don't immediately see a straightforward way to do it system wide (I guess a wrapper that straces init and children might do the trick but it's not particularly nice). Part of the trick for getting the general security win is to lower the barrier to entry.`
Sounds like something you could use tracepoints for maybe ? Failing that, kprobes ?
I'm pretty sure I've seen systemtap examples of this very thing years ago, but who knows if they even work any more.
Dave
Quoted 15 lines · Dave Jones
Quoted 6 lines · Mark Brown
Quoted 3 lines · Tony
Quoted 3 lines
It would be useful for the smaller build case to have a way of auditing which syscalls are actually in use on a system so you can then go through and construct a minimal config.
Quoted 7 lines
Quoted 1 line
"strace -c" ?
That works for specific processes but I don't immediately see a straightforward way to do it system wide (I guess a wrapper that straces init and children might do the trick but it's not particularly nice). Part of the trick for getting the general security win is to lower the barrier to entry.`
Quoted 6 lines
Sounds like something you could use tracepoints for maybe ? Failing that, kprobes ?
I'm pretty sure I've seen systemtap examples of this very thing years ago, but who knows if they even work any more.
It's actually pretty easy to do this with seccomp -- program it to send SIGSYS and watch the kernel logs. Admittedly, the lack of log + ENOSYS as a seccomp action might make this a little bit annoying.
--Andy
Quoted 5 lines
Sounds like something you could use tracepoints for maybe ? Failing that, kprobes ?
I'm pretty sure I've seen systemtap examples of this very thing years ago, but who knows if they even work any more.
If we do head to a world where systems are configured with a subset of system calls - it would be useful for application packages to come with a list of syscall dependencies. So you could avoid much sadness from installing a package that won't actually work.
But to get this right you'd need to do more than just strace/ kprobe/systemtap scan to see what they *typically* do. The system calls used will depend on the arguments given and the input data. E.g. a trivial test might conclude that bash(1) doesn't use the pipe(2) system call ... which it doesn't until some user types:
$ dmesg | grep ixgbe
-Tony
Quoted 12 lines · Tony
Quoted 5 lines
Sounds like something you could use tracepoints for maybe ? Failing that, kprobes ?
I'm pretty sure I've seen systemtap examples of this very thing years ago, but who knows if they even work any more.
If we do head to a world where systems are configured with a subset of system calls - it would be useful for application packages to come with a list of syscall dependencies. So you could avoid much sadness from installing a package that won't actually work.
This would also be useful for using seccomp to sandbox processes... to simply not let them do things they don't have a legitimate need to do.
For super-low-end embedded systems, it makes a lot of sense to be able to build a kernel with only the functionality needed by a small set of fixed applications (sometimes only one process which runs both as init and the application.)
-hpa
Quoted 10 lines · Dave Jones
Quoted 5 lines · Mark Brown
That works for specific processes but I don't immediately see a straightforward way to do it system wide (I guess a wrapper that straces init and children might do the trick but it's not particularly nice). Part of the trick for getting the general security win is to lower the barrier to entry.`
Sounds like something you could use tracepoints for maybe ? Failing that, kprobes ?
Tracepoints do run the risk of overflowing the buffer if run for too long but if it's the only thing running and/or is monitored that should be OK, it's more managable than strace. kprobes should definitely work I think if there's a suitably canned way of setting it up.
Quoted 6 lines · Mark Brown
Quoted 3 lines · Tony
Quoted 3 lines
It would be useful for the smaller build case to have a way of auditing which syscalls are actually in use on a system so you can then go through and construct a minimal config.
Quoted 7 lines
Quoted 1 line
"strace -c" ?
That works for specific processes but I don't immediately see a straightforward way to do it system wide (I guess a wrapper that straces init and children might do the trick but it's not particularly nice). Part of the trick for getting the general security win is to lower the barrier to entry.`
You can do it relatively easily with auditing, surely? Set up an audit rule for each syscall you aren't already sure is in use. Disable the rule when you see it used, and it shouldn't even have much of an overhead over and above what it takes to have auditing enabled in the first place (which we tried to keep to a minimum).
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 8 lines · David Woodhouse
Quoted 5 lines · Mark Brown
That works for specific processes but I don't immediately see a straightforward way to do it system wide (I guess a wrapper that straces init and children might do the trick but it's not particularly nice). Part of the trick for getting the general security win is to lower the barrier to entry.`
Quoted 5 lines
You can do it relatively easily with auditing, surely? Set up an audit rule for each syscall you aren't already sure is in use. Disable the rule when you see it used, and it shouldn't even have much of an overhead over and above what it takes to have auditing enabled in the first place (which we tried to keep to a minimum).
I suspect that's got too high a barrier to entry for a lot of users, especially since AFAICT it requires userspace tools on the target system. It should work though.
Quoted 19 lines · James Bottomley
Quoted 11 lines · Dave Jones
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernelSomething that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
My first reaction is reducing the attack surface sounds a reasonable idea. My second reaction is that the plural in options makes me want to run for the hills. Having a sea of options for enabling and disabling syscalls gives us the potential for having a set of kernels all with a slightly differing ABI as people choose what to enable and disable.
Quoted 8 lines
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel
CONFIG_SECURE_ROUTERwhich disables anything a secure router wouldn't need.For the distros we could have an ordinary and a reduced attack surface kernel
CONFIG_REDUCED_ATTACK_SURFACE.
Excellent idea. I'd like to be able to enable both options, for my company's routers but even on my home servers if possible (ie for a distribution which is tagged as 'server' distribution).
Guenter
Quoted 4 lines · On Fri, 02 May 2014 10:20:29 -0700 James Bottomley…
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel
CONFIG_SECURE_ROUTERwhich disables anything a secure router wouldn't need.
I was thinking the same thing.
Quoted 3 lines
For the distros we could have an ordinary and a reduced attack surface kernel
CONFIG_REDUCED_ATTACK_SURFACE.
Ug, that's a horrible name. Not selecting it would imply we want to increase the attack surface.
Quoted 3 lines
The thing I really want to avoid is binaries compiled for one distro not running on another because of syscall differences.
Agreed.
Your first config option name looks more the way we want to go. Didn't Linus once ask for config profiles? That is, we could say CONFIG_FIREWALL, and everything for a firewall would be set. Or CONFIG_LAPTOP, which would focus on everything for a laptop, etc.
What ever happened to that? The kbuild environment too scary for everyone?
I wounder if we should seek out people to rewrite it. Or at least document how the entire thing works. Every time I have to look at that code I get the willies.
-- Steve
Quoted 4 lines · James Bottomley
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel
CONFIG_SECURE_ROUTERwhich disables anything a secure router wouldn't need.
Have you seen the amount of stuff that OpenWRT packages? :)
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 7 lines · David Woodhouse
Quoted 4 lines · James Bottomley
If we do this, I think we should have a small number of options related to use case ... say something like a secure router kernel
CONFIG_SECURE_ROUTERwhich disables anything a secure router wouldn't need.Have you seen the amount of stuff that OpenWRT packages? :)
Yes ... I use it (and actually had to build it a while ago for one of my unsupported routers). *I* like running the kitchen sink from my router, but that's probably not the *normal* use, so CONFIG_SECURE_ROUTER would probably not be for general OpenWRT.
James
Quoted 19 lines · Dave Jones
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernelSomething that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
I first started thinking about this at LSF/MM where the subject of
sys_remap_file_pagescame up. "What even uses this?" "hardly anything". But for all the users that don't need it, there's this syscall always built in that does horrible things with VM internals. It's fortunate that there hasn't been anything particularly awful beyond simple DoS bugs inr_f_p.
Quoted 4 lines
Distribution kernels are in the sad position of having to always enable this stuff, but at least for people building their own kernels, or kernels for appliances, we could make their lives a little better by not even building this stuff in.
So I always thought various security modules or audit are there exactly
to limit attack surface like this (now please pardon my ignorance if I'm wrong because I know close to nothing about the security stuff). So in my imagination I'd say you could ship even a distro with a default policy where e.g. r_f_p would be prohibited and if you ever found an application that needs it, you could create a separate policy for it (and in the ideal case where the application is packaged by the distro the policy would come with it). Am I dreaming too much?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 35 lines · Dave Jones
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernelSomething that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
I first started thinking about this at LSF/MM where the subject of
sys_remap_file_pagescame up. "What even uses this?" "hardly anything". But for all the users that don't need it, there's this syscall always built in that does horrible things with VM internals. It's fortunate that there hasn't been anything particularly awful beyond simple DoS bugs inr_f_p.Distribution kernels are in the sad position of having to always enable this stuff, but at least for people building their own kernels, or kernels for appliances, we could make their lives a little better by not even building this stuff in.
I had a patch to make this particular syscall a
cond_syscall, but then XFS ate my homework and I haven't had chance to revisit this. So, my questions are:
- are there other obvious syscalls we could make optional without userspace freaking out when they suddenly start getting ENOSYS ?
- how much configurability here is too much ?
r_f_pwas an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..
Quoted 1 line
thoughts?
For deeply embedded, I think a good technique is to fine-tune the syscalls to the exact set of programs that will be run. For my size research last year (See http://elinux.org/System_Size_Auto-Reduction
and http://elinux.org/images/9/9e/Bird-Kernel-Size-Optimization-LCJ-2013.pdf), I discussed some results from doing the following:
- scanning the all the binaries in the file system
- generating a list of used and unused system calls
- adding a kernel mechanism to eliminate any unused syscalls, at compile time
(using LTO, all I had to do was make the syscall unreachable, and the compiler eliminated the calls automatically. Some of the work was removing parts of Andy Kleen's LTO patches which prevented unreferenced code from being optimized out.)
The syscalls were detected with a tool that scanned the program's assembly code on ARM, and found all syscall sequences. Binaries were statically linked for analysis.
On a default-configured kernel, the system eliminated 161 syscalls, and saved about 95k. On a minimally configured kernel, the system eliminated 120 syscalls, and saved 48K.
The remaining syscalls for my minimal system consisting of busybox and a web server was 184 syscalls. with additional coding and refactoring of the app and busybox, additional syscalls could have been eliminated.
Some syscalls that were unused, still ended up hanging around due to a few funny references. With some code refactoring, some additional savings might be possible.
Note that this system required no new kernel CONFIGs, and used macros to hide most of the complexity from the rest of the kernel source.
I did some other automatic code elimination techniques, with varying results. These can be seen in the presentation.
-- Tim
P.S. I realize this technique is not suitable for general-purpose distributions of Linux.
Quoted 4 lines · Tim
- adding a kernel mechanism to eliminate any unused syscalls, at compile time (using LTO, all I had to do was make the syscall unreachable, and the compiler eliminated the calls automatically. Some of the work was removing parts of Andy Kleen's LTO patches which prevented unreferenced code from being optimized out.)
There is some irony here.
-hpa
Quoted 11 lines · Dave Jones
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernelSomething that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
I'd certainly like to see just about every syscall made optional, for userspace that doesn't need it. For specialized systems, that certainly would decrease attack surface. However, seccomp decreases attack surface by the same amount, and for any except those specialized systems that would make more sense, because the set of available syscalls can then change with a simple policy change rather than a new kernel.
And this doesn't free us from the obligation to make all new APIs secure against hostile userspace.
Quoted 5 lines
I had a patch to make this particular syscall a
cond_syscall, but then XFS ate my homework and I haven't had chance to revisit this. So, my questions are:
- are there other obvious syscalls we could make optional without userspace freaking out when they suddenly start getting ENOSYS ?
I've attached a complete list of the syscalls from include/linux/syscalls.h that do not appear in kernel/sys_ni.c, and thus always exist. (syscalls.h notably does not include all the arch-specific syscalls, some of which might make sense to leave out as well.)
Of those, a few classes of syscalls that seem obvious, for various classes of specialized or legacy-free systems:
- For any syscall updated to have a foo2, foo3, etc, a single config option to leave out all the older versions would make sense, to go with userspace that never calls the older versions.
- Likewise, the non-64 file calls.
- Likewise, sys_old*
- splice/vmsplice/tee.
- sys_*sync*
- sys_clock_* and any other time functions.
- sys_sched_*
- All signal-related syscalls
- rlimit syscalls
- sys_*xattr*
- sys_nice
- sys_cap{get,set}
- fadvise, fallocate, readahead, etc.
- uid/gid functions.
- ioperm/iopl
- ptrace
- sendfile
- times
- utimes and company
Quoted 4 lines
- how much configurability here is too much ?
r_f_pwas an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..
We need a more systematic mechanism, I think. CONFIG_SYSCALL_FOO for every possible FOO seems too much, even for classes of syscalls. Ideally, we could feed in a table of syscalls collected by some analysis of the target userspace, and the kernel will then have exactly those syscalls.
- Josh Triplett
sys_accesssys_adjtimexsys_alarmsys_brksys_capgetsys_capsetsys_chdirsys_chmodsys_chownsys_chrootsys_clock_adjtimesys_clock_getressys_clock_gettimesys_clock_nanosleepsys_clock_settimesys_clonesys_closesys_creatsys_dupsys_dup2sys_dup3sys_execvesys_exitsys_exit_groupsys_faccessatsys_fadvise64sys_fadvise64_64sys_fallocatesys_fchdirsys_fchmodsys_fchmodatsys_fchownsys_fchownatsys_fcntlsys_fcntl64sys_fdatasyncsys_fgetxattrsys_flistxattrsys_forksys_fremovexattrsys_fsetxattrsys_fstatsys_fstat64sys_fstatat64sys_fstatfssys_fstatfs64sys_fsyncsys_ftruncatesys_ftruncate64sys_futimesatsys_getcpusys_getcwdsys_getdentssys_getdents64sys_getegidsys_geteuidsys_getgidsys_getgroupssys_gethostnamesys_getitimersys_getpgidsys_getpgrpsys_getpidsys_getppidsys_getprioritysys_getresgidsys_getresuidsys_getrlimitsys_getrusagesys_getsidsys_gettidsys_gettimeofdaysys_getuidsys_getxattrsys_ioctlsys_iopermsys_killsys_lchownsys_lgetxattrsys_linksys_linkatsys_listxattrsys_llistxattrsys_llseeksys_lremovexattrsys_lseeksys_lsetxattrsys_lstatsys_lstat64sys_mkdirsys_mkdiratsys_mknodsys_mknodatsys_mmap_pgoffsys_mountsys_munmapsys_nanosleepsys_newfstatsys_newfstatatsys_newlstatsys_newstatsys_newunamesys_ni_syscallsys_nicesys_old_getrlimitsys_old_mmapsys_old_readdirsys_old_selectsys_oldumountsys_oldunamesys_opensys_openatsys_pausesys_personalitysys_pipesys_pipe2sys_pivot_rootsys_pollsys_ppollsys_prctlsys_pread64sys_preadvsys_prlimit64sys_pselect6sys_ptracesys_pwrite64sys_pwritevsys_readsys_readaheadsys_readlinksys_readlinkatsys_readvsys_rebootsys_removexattrsys_renamesys_renameatsys_renameat2sys_restart_syscallsys_rmdirsys_rt_sigactionsys_rt_sigpendingsys_rt_sigprocmasksys_rt_sigqueueinfosys_rt_sigsuspendsys_rt_sigtimedwaitsys_rt_tgsigqueueinfosys_sched_get_priority_maxsys_sched_get_priority_minsys_sched_getaffinitysys_sched_getattrsys_sched_getparamsys_sched_getschedulersys_sched_rr_get_intervalsys_sched_setaffinitysys_sched_setattrsys_sched_setparamsys_sched_setschedulersys_sched_yieldsys_selectsys_sendfilesys_sendfile64sys_set_tid_addresssys_setdomainnamesys_setfsgidsys_setfsuidsys_setgidsys_setgroupssys_sethostnamesys_setitimersys_setnssys_setpgidsys_setprioritysys_setregidsys_setresgidsys_setresuidsys_setreuidsys_setrlimitsys_setsidsys_settimeofdaysys_setuidsys_setxattrsys_sgetmasksys_sigactionsys_sigaltstacksys_signalsys_sigpendingsys_sigprocmasksys_sigsuspendsys_splicesys_ssetmasksys_statsys_stat64sys_statfssys_statfs64sys_stimesys_symlinksys_symlinkatsys_syncsys_sync_file_rangesys_sync_file_range2sys_syncfssys_sysctlsys_sysinfosys_teesys_tgkillsys_timesys_timer_createsys_timer_deletesys_timer_getoverrunsys_timer_gettimesys_timer_settimesys_timessys_tkillsys_truncatesys_truncate64sys_umasksys_umountsys_unamesys_unlinksys_unlinkatsys_unsharesys_ustatsys_utimesys_utimensatsys_utimessys_vforksys_vhangupsys_vmsplicesys_wait4sys_waitidsys_waitpidsys_writesys_writev
Quoted 66 lines · Josh Triplett
Quoted 11 lines · Dave Jones
Quoted 4 lines · Josh Triplett
Topics:
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernelSomething that's partially related here: Making stuff optional reduces attack surface the kernel presents. We're starting to grow more and more CONFIG options to disable syscalls. I'd like to hear peoples reactions on introducing even more optionality in this area.
I'd certainly like to see just about every syscall made optional, for userspace that doesn't need it. For specialized systems, that certainly would decrease attack surface. However, seccomp decreases attack surface by the same amount, and for any except those specialized systems that would make more sense, because the set of available syscalls can then change with a simple policy change rather than a new kernel.
And this doesn't free us from the obligation to make all new APIs secure against hostile userspace.
Quoted 5 lines
I had a patch to make this particular syscall a
cond_syscall, but then XFS ate my homework and I haven't had chance to revisit this. So, my questions are:
- are there other obvious syscalls we could make optional without userspace freaking out when they suddenly start getting ENOSYS ?I've attached a complete list of the syscalls from
include/linux/syscalls.hthat do not appear inkernel/sys_ni.c, and thus always exist. (syscalls.hnotably does not include all the arch-specific syscalls, some of which might make sense to leave out as well.)Of those, a few classes of syscalls that seem obvious, for various classes of specialized or legacy-free systems:
- For any syscall updated to have a foo2, foo3, etc, a single config option to leave out all the older versions would make sense, to go with userspace that never calls the older versions.
- Likewise, the non-64 file calls.
- Likewise, sys_old*
- splice/vmsplice/tee.
- sys_*sync*
- sys_clock_* and any other time functions.
- sys_sched_*
- All signal-related syscalls
- rlimit syscalls
- sys_*xattr*
-sys_nice
- sys_cap{get,set}
- fadvise, fallocate, readahead, etc.
- uid/gid functions.
- ioperm/iopl
- ptrace
- sendfile
- times
- utimes and companyQuoted 4 lines
- how much configurability here is too much ?
r_f_pwas an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..
Quoted 5 lines
We need a more systematic mechanism, I think.
CONFIG_SYSCALL_FOOfor every possible FOO seems too much, even for classes of syscalls. Ideally, we could feed in a table of syscalls collected by some analysis of the target userspace, and the kernel will then have exactly those syscalls.
In my system, I set it up so that every syscall had it's own SYSCALL_DEFINE macro. and then used a single header file consisting of lines like:
#define syscall_setreuid16_unused 1
The SYSCALL_DEFINE macros would then control whether the syscall was extern'ed or not. A separate mechanism converted the CALL macro in calls.S (on ARM) to use sys_ni_syscall, and LTO made the (now unreferenced) function evaporate.
Overall, this allowed control of every syscall with a single easily generated (or easily hand-edited) header file. And, with a stub header file, everything worked as without the changes.
The header file was auto-generated by tools that scanned the user-space programs for all possible syscall sequences.
In hindsight this system could probably be improved with some extra tweaking to the base SYSCALL_DEFINE macros, to make
it so no source changes were required at the function definition sites.
In any event, it's possible to get per-syscall granularity without having to add new CONFIGS (but at the expense of adding a generated header file).
-- Tim
Quoted 28 lines · Tim
Quoted 7 lines · Josh Triplett
Quoted 4 lines · Dave Jones
- how much configurability here is too much ?
r_f_pwas an obvious candidate because it's.. well, nasty. Some of the more straightforward syscalls may not be such a big deal, but then we have CONFIG's for kcmp and other 'simple' syscalls already..Quoted 5 lines
We need a more systematic mechanism, I think.
CONFIG_SYSCALL_FOOfor every possible FOO seems too much, even for classes of syscalls. Ideally, we could feed in a table of syscalls collected by some analysis of the target userspace, and the kernel will then have exactly those syscalls.In my system, I set it up so that every syscall had it's own
SYSCALL_DEFINEmacro. and then used a single header file consisting of lines like:
#definesyscall_setreuid16_unused1The
SYSCALL_DEFINEmacros would then control whether the syscall was extern'ed or not. A separate mechanism converted the CALL macro incalls.S(on ARM) to usesys_ni_syscall, and LTO made the (now unreferenced) function evaporate.Overall, this allowed control of every syscall with a single easily generated (or easily hand-edited) header file. And, with a stub header file, everything worked as without the changes.
Quoted 6 lines
The header file was auto-generated by tools that scanned the user-space programs for all possible syscall sequences.
In hindsight this system could probably be improved with some extra tweaking to the base
SYSCALL_DEFINEmacros, to make
it so no source changes were required at the function definition sites.
Another possibility: make all the syscall functions garbage-collectable, and only keep those referenced from the actual syscall table. Then generate the syscall table accordingly.
- Josh Triplett
Quoted 2 lines · Josh Triplett
- An overview of why the kernel's size still matters today ("but don't we all have tons of memory and storage?")
Kernel size matters quite a bit for performance. Processor caches are key to performance and therefore the cache footprint of a function determines the the possible performance. The smaller the functions and the less data they access the faster they will run.
Therefore it needs to be possible to reduce the size of the kernel by disabling unwanted functionality (f.e. cgroups). In order for that to happen features need to be as independent as possible and also the user space tools (like systemd) need to be able to handle a kernel with reduced functionality.
Quoted 5 lines
- Tiny in RAM versus tiny on storage.
- How much the kernel has grown over time.
- How size regressions happen and how to avoid them
- Size measurement, bloat-o-meter, allnoconfig, and other tools
Besides the discussions about keeping the kernel size small by deselecting features, I believe we have a few options left to reduce the size just by rethinking how data is arranged. I have just started to research 'strings' in the kernel and am already seeing patterns which look like low hanging fruits to me. (Unsurprisingly, given the amount of copy&paste code.) Take the OOM message removal as one example, such things can be fixed and prevented. Although my focus is 'strings' currently, I am sure the lessons learned can be of generic interest. So to say: keep the bloat on a level that is really needed for a new feature.
I proposed to present my results at LinuxCon anyhow, so that would fit.
Disclaimer: Yes, I work more on drivers than core code :)
Quoted 10 lines · Josh Triplett
Over time, the Linux kernel has grown far more featureful, but it has also grown significantly larger, even with all the optional features turned off. For the last several years, I've been working to make the kernel smaller, and mentoring/coordinating projects to do the same, to enable ridiculously small embedded applications and other fun uses. I'd like to discuss that work at Kernel Summit, get size regressions on the radar of kernel developers and subsystem maintainers, and solicit discussion on future possibilities to shrink the kernel further.
Quoted 15 lines
Topics:
- An overview of why the kernel's size still matters today ("but don't we all have tons of memory and storage?")
- Tiny in RAM versus tiny on storage.
- How much the kernel has grown over time.
- How size regressions happen and how to avoid them
- Size measurement, bloat-o-meter, allnoconfig, and other tools
- Compression and the decompression stub
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernel
- Discussion on proposed ways to make the kernel tiny, how much they might save, how much work they'd require, and how to implement them with minimal impact to the un-shrunken common case.
I'd really like to see a discussion of mechanisms to improve automated reduction of the kernel. This will really help, IMHO, to avoid excessive configurability, and hopefully ameliorate complaints about the long-term maintenance cost of keeping small configurations available in-tree.
One prime example of this would be the "static-ification" of DT, e.g. replacing calls to lookup DT info with constants (via macros or some other source replacement trick), so that we can leverage the compiler's optimizations for constant propagation and dead code removal.
Quoted 6 lines
After the session, I'll prepare and maintain a detailed summary of the proposed ideas, ordered by how much space they'd save versus how much work they'd be. I plan to maintain that list on an ongoing basis, to coordinate tinification projects for ongoing work by people working on embedded applications, and for the benefit of mentorship projects such as OPW and SoC.
Thanks for taking the lead on this!
Can I recommend we use the linux-embedded mailing list for discussions? It's underutilized and this topic seems like a good fit. Also, the elinux wiki is available if you're looking for a place to maintain information on this. The Linux-tiny material there is stale, but I have been thinking about updating it since the last ELC.
-- Tim
Quoted 27 lines · Tim
Quoted 10 lines · Josh Triplett
Over time, the Linux kernel has grown far more featureful, but it has also grown significantly larger, even with all the optional features turned off. For the last several years, I've been working to make the kernel smaller, and mentoring/coordinating projects to do the same, to enable ridiculously small embedded applications and other fun uses. I'd like to discuss that work at Kernel Summit, get size regressions on the radar of kernel developers and subsystem maintainers, and solicit discussion on future possibilities to shrink the kernel further.
Quoted 15 lines
Topics:
- An overview of why the kernel's size still matters today ("but don't we all have tons of memory and storage?")
- Tiny in RAM versus tiny on storage.
- How much the kernel has grown over time.
- How size regressions happen and how to avoid them
- Size measurement, bloat-o-meter, allnoconfig, and other tools
- Compression and the decompression stub
-Kconfig, and avoiding excessive configurability in the pursuit of tiny
- Optimizing a kernel for its exact target userspace.
- Examples of shrinking the kernel
- Discussion on proposed ways to make the kernel tiny, how much they might save, how much work they'd require, and how to implement them with minimal impact to the un-shrunken common case.
Quoted 4 lines
I'd really like to see a discussion of mechanisms to improve automated reduction of the kernel. This will really help, IMHO, to avoid excessive configurability, and hopefully ameliorate complaints about the long-term maintenance cost of keeping small configurations available in-tree.
Agreed; that's one of the things I planned to include, under "avoiding excessive configurability" and "Optimizing a kernel for its exact target userspace". LTO is another big part of this.
Quoted 4 lines
One prime example of this would be the "static-ification" of DT, e.g. replacing calls to lookup DT info with constants (via macros or some other source replacement trick), so that we can leverage the compiler's optimizations for constant propagation and dead code removal.
I agree completely. I'd also like to optimize the kernel for a specific kernel command line (turning global variables set from the command line into static consts), automatic syscall elimination (as discussed elsewhere in the thread), and even optimization for specific target hardware configurations (not just DT, but bits normally found via probing).
Quoted 7 lines
Quoted 6 lines
After the session, I'll prepare and maintain a detailed summary of the proposed ideas, ordered by how much space they'd save versus how much work they'd be. I plan to maintain that list on an ongoing basis, to coordinate tinification projects for ongoing work by people working on embedded applications, and for the benefit of mentorship projects such as OPW and SoC.
Quoted 7 lines
Thanks for taking the lead on this!
Can I recommend we use the linux-embedded mailing list for discussions? It's underutilized and this topic seems like a good fit. Also, the elinux wiki is available if you're looking for a place to maintain information on this. The Linux-tiny material there is stale, but I have been thinking about updating it since the last ELC.
Sounds plausible to me. And I've been seriously considering taking on maintenance of a new tiny tree.
- Josh Triplett
Quoted 4 lines
One prime example of this would be the "static-ification" of DT, e.g. replacing calls to lookup DT info with constants (via macros or some other source replacement trick), so that we can leverage the compiler's optimizations for constant propagation and dead code removal.
Could Coccinelle help with this somehow? The requirement would be that there is some pattern that can be recognized, and the desired change is static, ie a modification in the source code.
julia
[Ksummit-discuss] tiny.wiki.kernel.org
In reply to Josh TriplettIn preparation for the tinification discussion at Kernel Summit, I've created tiny.wiki.kernel.org to track the status of kernel tinification, potential projects, and HOWTOs.
Please feel free to add ideas or other material there in advance of the discussion on Monday.
- Josh Triplett