Re-tune x86 uaccess code for PREEMPT_VOLUNTARY
Subsystems (62)
The x86 user access functions (*_user) were originally very well tuned, with partial inline code and other optimizations.
Then over time various new checks -- particularly the sleep checks for a voluntary preempt kernel -- destroyed a lot of the tunings
A typical user access operation is now doing multiple useless function calls. Also the without force inline gcc's inlining policy makes it even worse, with adding more unnecessary calls.
Here's a typical example from ftrace:
10) | might_fault() {
10) | _cond_resched() {
10) | should_resched() {
10) | need_resched() {
10) 0.063 us | test_ti_thread_flag();
10) 0.643 us | }
10) 1.238 us | }
10) 1.845 us | }
10) 2.438 us | }So we spent 2.5us doing nothing (ok it's a bit less without ftrace, but still pretty bad)
Then in other cases we would have an out of line function, but would actually do the might_sleep() checks in the inlined caller. This doesn't make any sense at all.
There were also a few other problems, for example the x86-64 uaccess code regularly falls back to string functions, even though a simple mov would be enough. For example every futex access to the lock variable would actually use string instructions, even though it's just 4 bytes.
This patch kit is an attempt to get us back to sane code, mostly by doing proper inlining and doing sleep checks in the right place. Unfortunately I had to add one tree sweep to avoid an nasty include loop.
It costs a bit of text space, but I think it's worth it (if only to keep my blood pressure down while reading ftrace logs...)
I haven't done any particular benchmarks, but important low level functions just ought to be fast.
64bit:
13249492 1881328 1159168 16289988 f890c4 vmlinux-before-uaccess 13260877 1877232 1159168 16297277 f8ad3d vmlinux-uaccess
+ 11k, +0.08%
32bit:
11223248 899512 1916928 14039688 d63a88 vmlinux-before-uaccess 11230358 895416 1916928 14042702 d6464e vmlinux-uaccess
+ 7k, +0.06%
PATCH 1/13 x86: Add 1/2/4/8 byte optimization to 64bit __copy_{from,to}_user_inatomic
From: Andi Kleen <[email protected]>
The 64bit __copy_{from,to}_user_inatomic always called copy_from_user_generic, but skipped the special optimizations for 1/2/4/8 byte accesses.
This especially hurts the futex call, which accesses the 4 byte futex user value with a complicated fast string operation in a function call, instead of a single movl.
Use __copy_{from,to}_user for _inatomic instead to get the same optimizations. The only problem was the might_fault() in those functions. So move that to new wrapper and call __copy_{f,t}_user_nocheck() from *_inatomic directly.
32bit already did this correctly by duplicating the code.
--- arch/x86/include/asm/uaccess_64.h | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)
arch/x86/include/asm/uaccess_64.h +18 -6
@@ -77,11 +77,10 @@ int copy_to_user(void __user *dst, const void *src, unsigned size) }static __always_inline __must_check -int __copy_from_user(void *dst, const void __user *src, unsigned size) +int __copy_from_user_nocheck(void *dst, const void __user *src, unsigned size) {int ret =0 ; -might_fault ();if (!__builtin_constant_p (size))return copy_user_generic (dst, (__force void *)src, size);switch (size) {
@@ -121,11 +120,17 @@ int __copy_from_user(void *dst, const void __user *src, unsigned size) }static __always_inline __must_check -int __copy_to_user(void __user *dst, const void *src, unsigned size) +int __copy_from_user(void *dst, const void __user *src, unsigned size) +{ +might_fault (); +return __copy_from_user_nocheck (dst, src, size); +} + +static __always_inline __must_check +int __copy_to_user_nocheck(void __user *dst, const void *src, unsigned size) {int ret =0 ; -might_fault ();if (!__builtin_constant_p (size))return copy_user_generic ((__force void *)dst, src, size);switch (size) {
@@ -165,6 +170,13 @@ int __copy_to_user(void __user *dst, const void *src, unsigned size) }static __always_inline __must_check +int __copy_to_user(void __user *dst, const void *src, unsigned size) +{ +might_fault (); +return __copy_to_user_nocheck (dst, src, size); +} + +static __always_inline __must_check int __copy_in_user(void __user *dst, const void __user *src, unsigned size) {int ret =0 ;
@@ -220,13 +232,13 @@ int __copy_in_user(void __user *dst, const void __user *src, unsigned size) static __must_check __always_inline int__copy_from_user_inatomic (void *dst,const void __user *src,unsigned size) { -return copy_user_generic (dst, (__force const void *)src, size); +return __copy_from_user_nocheck (dst, (__force const void *)src, size); } static __must_check __always_inline int__copy_to_user_inatomic (void __user *dst,const void *src,unsigned size) { -return copy_user_generic ((__force void *)dst, src, size); +return __copy_to_user_nocheck ((__force void *)dst, src, size); } extern long __copy_user_nocache(void *dst, const void __user *src,
Signature · 1 line
1.8.3.1
PATCH 2/13 x86: Include linux/sched.h in asm/uaccess.h
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
uaccess.h uses might_sleep, but there is currently no explicit include for this. Since a upcoming patch moves might_sleep into sched.h include sched.h here.
--- arch/x86/include/asm/uaccess.h | 1 + 1 file changed, 1 insertion(+)
arch/x86/include/asm/uaccess.h +1 -0
@@ -3,6 +3,7 @@/* * User space memory access functions */ +#include <linux/sched.h> #include <linux/errno.h> #include <linux/compiler.h> #include <linux/thread_info.h>
Signature · 1 line
1.8.3.1
PATCH 3/13 tree-sweep: Include linux/sched.h for might_sleep users
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
might_sleep is moving from linux/kernel.h to linux/sched.h, so any users need to include linux/sched.h
This was done with a mechanistic script and some uses may be redundant (already included in some other include file). However it's good practice to always include any needed symbols from the top level .c file.
Tested with x86-64 allyesconfig. I used to do a x86-32 allyesconfig on a old kernel, but since that is broken now I didn't retest.
--- arch/arm/common/mcpm_entry.c | 1 + arch/arm/mach-omap2/omap_hwmod.c | 1 + arch/arm/mm/highmem.c | 1 + arch/blackfin/kernel/bfin_gpio.c | 1 + arch/frv/mm/highmem.c | 1 + arch/m32r/include/asm/uaccess.h | 1 + arch/microblaze/include/asm/highmem.h | 1 + arch/mn10300/include/asm/uaccess.h | 1 + arch/parisc/include/asm/cacheflush.h | 1 + arch/powerpc/include/asm/highmem.h | 1 + arch/powerpc/kernel/rtas.c | 1 + arch/powerpc/lib/checksum_wrappers_64.c | 1 + arch/powerpc/lib/usercopy_64.c | 1 + arch/tile/mm/highmem.c | 1 + arch/x86/include/asm/checksum_32.h | 1 + arch/x86/lib/csum-wrappers_64.c | 1 + arch/x86/mm/highmem_32.c | 1 + arch/x86/mm/mmio-mod.c | 1 + block/blk-cgroup.c | 1 + block/blk-core.c | 1 + block/genhd.c | 1 + drivers/base/dma-buf.c | 1 + drivers/block/rsxx/dev.c | 1 + drivers/dma/ipu/ipu_irq.c | 1 + drivers/gpio/gpiolib.c | 1 + drivers/ide/ide-io.c | 1 + drivers/infiniband/hw/amso1100/c2_cq.c | 1 + drivers/infiniband/hw/cxgb3/iwch_cm.c | 1 + drivers/infiniband/hw/cxgb4/cm.c | 1 + drivers/md/dm.c | 1 + drivers/md/raid5.c | 1 + drivers/mmc/core/core.c | 1 + drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c | 1 + drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c | 1 + drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.h | 1 + drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 1 + drivers/net/ethernet/intel/igbvf/netdev.c | 1 + drivers/net/ethernet/sfc/falcon.c | 1 + drivers/net/ieee802154/at86rf230.c | 1 + drivers/net/ieee802154/fakelb.c | 1 + drivers/net/wireless/ath/carl9170/usb.c | 1 + drivers/net/wireless/ath/wil6210/wmi.c | 1 + drivers/net/wireless/b43/dma.c | 1 + drivers/net/wireless/b43/main.c | 1 + drivers/net/wireless/b43/phy_a.c | 1 + drivers/net/wireless/b43/phy_g.c | 1 + drivers/net/wireless/b43legacy/dma.c | 1 + drivers/net/wireless/b43legacy/radio.c | 1 + drivers/net/wireless/cw1200/cw1200_spi.c | 1 + drivers/net/wireless/iwlwifi/dvm/sta.c | 1 + drivers/net/wireless/iwlwifi/iwl-op-mode.h | 1 + drivers/net/wireless/iwlwifi/iwl-trans.h | 1 + drivers/net/wireless/libertas_tf/cmd.c | 1 + drivers/pci/iov.c | 1 + drivers/pci/pci.c | 1 + drivers/platform/olpc/olpc-ec.c | 1 + drivers/ssb/driver_pcicore.c | 1 + drivers/staging/lustre/lustre/llite/remote_perm.c | 1 + drivers/staging/lustre/lustre/obdclass/cl_lock.c | 1 + drivers/staging/lustre/lustre/obdclass/cl_object.c | 1 + drivers/staging/lustre/lustre/obdclass/cl_page.c | 1 + drivers/staging/lustre/lustre/osc/osc_lock.c | 1 + drivers/staging/lustre/lustre/osc/osc_page.c | 1 + drivers/staging/lustre/lustre/ptlrpc/client.c | 1 + drivers/staging/lustre/lustre/ptlrpc/gss/gss_cli_upcall.c | 1 + drivers/staging/lustre/lustre/ptlrpc/gss/gss_pipefs.c | 1 + drivers/staging/lustre/lustre/ptlrpc/sec.c | 1 + drivers/staging/lustre/lustre/ptlrpc/sec_config.c | 1 + drivers/staging/lustre/lustre/ptlrpc/sec_gc.c | 1 + drivers/usb/core/hcd.c | 1 + drivers/usb/core/urb.c | 1 + drivers/video/atmel_lcdfb.c | 1 + fs/block_dev.c | 1 + fs/buffer.c | 1 + fs/dcache.c | 1 + fs/ext3/inode.c | 1 + fs/ext4/ext4_jbd2.c | 1 + fs/ext4/inode.c | 1 + fs/ext4/mballoc.c | 1 + fs/file_table.c | 1 + fs/inode.c | 1 + fs/jbd/revoke.c | 1 + fs/jbd2/revoke.c | 1 + fs/locks.c | 1 + fs/nfs/nfs4filelayoutdev.c | 1 + fs/nfs/nfs4proc.c | 1 + fs/nfs/nfs4state.c | 1 + fs/nilfs2/the_nilfs.c | 1 + fs/xfs/xfs_mount.c | 1 + include/asm-generic/gpio.h | 1 + include/linux/buffer_head.h | 1 + include/linux/clk.h | 1 + include/linux/gpio.h | 1 + include/linux/highmem.h | 1 + include/linux/pagemap.h | 1 + kernel/fork.c | 1 + kernel/freezer.c | 1 + kernel/irq/chip.c | 1 + kernel/nsproxy.c | 1 + kernel/printk/printk.c | 1 + kernel/sched/core.c | 1 + kernel/smp.c | 1 + mm/hugetlb.c | 1 + mm/memory.c | 1 + mm/mmap.c | 1 + mm/rmap.c | 1 + net/caif/cfcnfg.c | 1 + net/mac80211/driver-ops.h | 1 + net/mac80211/key.c | 1 + net/mac80211/main.c | 1 + net/mac80211/sta_info.c | 1 + net/phonet/pep.c | 1 + net/sunrpc/clnt.c | 1 + net/wimax/op-msg.c | 1 + net/wimax/op-reset.c | 1 + net/wimax/op-rfkill.c | 1 + net/wireless/wext-proc.c | 1 + sound/core/info.c | 1 + virt/kvm/async_pf.c | 1 + 120 files changed, 120 insertions(+)
arch/arm/common/mcpm_entry.c +1 -0
@@ -10,6 +10,7 @@*/ #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/init.h> #include <linux/irqflags.h>
arch/arm/mach-omap2/omap_hwmod.c +1 -0
@@ -127,6 +127,7 @@*/ #undef DEBUG +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/errno.h> #include <linux/io.h>
arch/arm/mm/highmem.c +1 -0
@@ -10,6 +10,7 @@* published by the Free Software Foundation. */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/highmem.h> #include <linux/interrupt.h>
arch/blackfin/kernel/bfin_gpio.c +1 -0
@@ -6,6 +6,7 @@* Licensed under the GPL-2 or later */ +#include <linux/sched.h> #include <linux/delay.h> #include <linux/module.h> #include <linux/err.h>
arch/frv/mm/highmem.c +1 -0
@@ -8,6 +8,7 @@* as published by the Free Software Foundation; either version * 2 of the License, or (at your option) any later version. */ +#include <linux/sched.h> #include <linux/highmem.h> #include <linux/module.h>
arch/m32r/include/asm/uaccess.h +1 -0
@@ -11,6 +11,7 @@/* * User space memory access functions */ +#include <linux/sched.h> #include <linux/errno.h> #include <linux/thread_info.h> #include <asm/page.h>
arch/microblaze/include/asm/highmem.h +1 -0
@@ -20,6 +20,7 @@#ifdef __KERNEL__#include <linux/init.h> +#include <linux/sched.h> #include <linux/interrupt.h> #include <linux/uaccess.h> #include <asm/fixmap.h>
arch/mn10300/include/asm/uaccess.h +1 -0
@@ -14,6 +14,7 @@/* * User space memory access functions */ +#include <linux/sched.h> #include <linux/thread_info.h> #include <linux/kernel.h> #include <asm/page.h>
arch/parisc/include/asm/cacheflush.h +1 -0
@@ -2,6 +2,7 @@#define _PARISC_CACHEFLUSH_H#include <linux/mm.h> +#include <linux/sched.h> #include <linux/uaccess.h> #include <asm/tlbflush.h>
arch/powerpc/include/asm/highmem.h +1 -0
@@ -22,6 +22,7 @@#ifdef __KERNEL__ +#include <linux/sched.h> #include <linux/interrupt.h> #include <asm/kmap_types.h> #include <asm/tlbflush.h>
arch/powerpc/kernel/rtas.c +1 -0
@@ -11,6 +11,7 @@* 2 of the License, or (at your option) any later version. */ +#include <linux/sched.h> #include <stdarg.h> #include <linux/kernel.h> #include <linux/types.h>
arch/powerpc/lib/checksum_wrappers_64.c +1 -0
@@ -17,6 +17,7 @@* * Author: Anton Blanchard <[email protected]> */ +#include <linux/sched.h> #include <linux/export.h> #include <linux/compiler.h> #include <linux/types.h>
arch/powerpc/lib/usercopy_64.c +1 -0
@@ -6,6 +6,7 @@* as published by the Free Software Foundation; either version * 2 of the License, or (at your option) any later version. */ +#include <linux/sched.h> #include <linux/module.h> #include <asm/uaccess.h>
arch/tile/mm/highmem.c +1 -0
@@ -12,6 +12,7 @@* more details. */ +#include <linux/sched.h> #include <linux/highmem.h> #include <linux/module.h> #include <linux/pagemap.h>
arch/x86/include/asm/checksum_32.h +1 -0
@@ -2,6 +2,7 @@#define _ASM_X86_CHECKSUM_32_H#include <linux/in6.h> +#include <linux/sched.h> #include <asm/uaccess.h>
arch/x86/lib/csum-wrappers_64.c +1 -0
@@ -4,6 +4,7 @@* * Wrappers of assembly checksum functions for x86-64. */ +#include <linux/sched.h> #include <asm/checksum.h> #include <linux/module.h>
arch/x86/mm/highmem_32.c +1 -0
@@ -1,3 +1,4 @@ +#include <linux/sched.h> #include <linux/highmem.h> #include <linux/module.h> #include <linux/swap.h> /* for totalram_pages */
arch/x86/mm/mmio-mod.c +1 -0
@@ -24,6 +24,7 @@#define DEBUG 1 +#include <linux/sched.h> #include <linux/module.h> #include <linux/debugfs.h> #include <linux/slab.h>
block/blk-cgroup.c +1 -0
@@ -11,6 +11,7 @@* Nauman Rafique <[email protected]> */ #include <linux/ioprio.h> +#include <linux/sched.h> #include <linux/kdev_t.h> #include <linux/module.h> #include <linux/err.h>
block/blk-core.c +1 -0
@@ -11,6 +11,7 @@/* * This handles all read/write requests to block devices */ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/backing-dev.h>
block/genhd.c +1 -0
@@ -2,6 +2,7 @@* gendisk handling */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/fs.h> #include <linux/genhd.h>
drivers/base/dma-buf.c +1 -0
@@ -22,6 +22,7 @@* this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <linux/sched.h> #include <linux/fs.h> #include <linux/slab.h> #include <linux/dma-buf.h>
drivers/block/rsxx/dev.c +1 -0
@@ -23,6 +23,7 @@*/ #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/interrupt.h> #include <linux/module.h> #include <linux/pci.h>
drivers/dma/ipu/ipu_irq.c +1 -0
@@ -7,6 +7,7 @@* published by the Free Software Foundation. */ +#include <linux/sched.h> #include <linux/init.h> #include <linux/err.h> #include <linux/spinlock.h>
drivers/gpio/gpiolib.c +1 -0
@@ -1,3 +1,4 @@ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/interrupt.h>
drivers/ide/ide-io.c +1 -0
@@ -24,6 +24,7 @@*/ +#include <linux/sched.h> #include <linux/module.h> #include <linux/types.h> #include <linux/string.h>
drivers/infiniband/hw/amso1100/c2_cq.c +1 -0
@@ -35,6 +35,7 @@* SOFTWARE. * */ +#include <linux/sched.h> #include <linux/gfp.h> #include "c2.h"
drivers/infiniband/hw/cxgb3/iwch_cm.c +1 -0
@@ -29,6 +29,7 @@* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/list.h> #include <linux/slab.h>
drivers/infiniband/hw/cxgb4/cm.c +1 -0
@@ -29,6 +29,7 @@* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/list.h> #include <linux/workqueue.h>
drivers/md/dm.c +1 -0
@@ -5,6 +5,7 @@* This file is released under the GPL. */ +#include <linux/sched.h> #include "dm.h" #include "dm-uevent.h"
drivers/md/raid5.c +1 -0
@@ -43,6 +43,7 @@* miss any bits. */ +#include <linux/sched.h> #include <linux/blkdev.h> #include <linux/kthread.h> #include <linux/raid/pq.h>
drivers/mmc/core/core.c +1 -0
@@ -10,6 +10,7 @@* it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/init.h> #include <linux/interrupt.h>
drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +1 -0
@@ -17,6 +17,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/sched.h> #include <linux/module.h> #include <linux/moduleparam.h> #include <linux/kernel.h>
drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c +1 -0
@@ -19,6 +19,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/sched.h> #include <linux/module.h> #include <linux/crc32.h> #include <linux/netdevice.h>
drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.h +1 -0
@@ -20,6 +20,7 @@#define BNX2X_SRIOV_H #include "bnx2x_vfpf.h" +#include <linux/sched.h> #include "bnx2x.h" enum sample_bulletin_result {
drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c +1 -0
@@ -17,6 +17,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/sched.h> #include "bnx2x_stats.h" #include "bnx2x_cmn.h" #include "bnx2x_sriov.h"
drivers/net/ethernet/intel/e1000e/netdev.c +1 -0
@@ -28,6 +28,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/sched.h> #include <linux/module.h> #include <linux/types.h> #include <linux/init.h>
drivers/net/ethernet/intel/igbvf/netdev.c +1 -0
@@ -27,6 +27,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/sched.h> #include <linux/module.h> #include <linux/types.h> #include <linux/init.h>
drivers/net/ethernet/sfc/falcon.c +1 -0
@@ -8,6 +8,7 @@* by the Free Software Foundation, incorporated herein by reference. */ +#include <linux/sched.h> #include <linux/bitops.h> #include <linux/delay.h> #include <linux/pci.h>
drivers/net/ieee802154/at86rf230.c +1 -0
@@ -21,6 +21,7 @@* Alexander Smirnov <[email protected]> */ #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/module.h> #include <linux/interrupt.h> #include <linux/gpio.h>
drivers/net/ieee802154/fakelb.c +1 -0
@@ -23,6 +23,7 @@*/ #include <linux/module.h> +#include <linux/sched.h> #include <linux/timer.h> #include <linux/platform_device.h> #include <linux/netdevice.h>
drivers/net/wireless/ath/carl9170/usb.c +1 -0
@@ -37,6 +37,7 @@* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/slab.h> #include <linux/usb.h>
drivers/net/wireless/ath/wil6210/wmi.c +1 -0
@@ -15,6 +15,7 @@*/ #include <linux/etherdevice.h> +#include <linux/sched.h> #include <linux/if_arp.h> #include "wil6210.h"
drivers/net/wireless/b43/dma.c +1 -0
@@ -27,6 +27,7 @@*/ +#include <linux/sched.h> #include "b43.h" #include "dma.h" #include "main.h"
drivers/net/wireless/b43/main.c +1 -0
@@ -32,6 +32,7 @@*/ +#include <linux/sched.h> #include <linux/delay.h> #include <linux/init.h> #include <linux/module.h>
drivers/net/wireless/b43/phy_a.c +1 -0
@@ -26,6 +26,7 @@*/ +#include <linux/sched.h> #include <linux/slab.h> #include "b43.h"
drivers/net/wireless/b43/phy_g.c +1 -0
@@ -26,6 +26,7 @@*/ +#include <linux/sched.h> #include "b43.h" #include "phy_g.h" #include "phy_common.h"
drivers/net/wireless/b43legacy/dma.c +1 -0
@@ -27,6 +27,7 @@*/ +#include <linux/sched.h> #include "b43legacy.h" #include "dma.h" #include "main.h"
drivers/net/wireless/b43legacy/radio.c +1 -0
@@ -29,6 +29,7 @@*/ +#include <linux/sched.h> #include <linux/delay.h> #include "b43legacy.h"
drivers/net/wireless/cw1200/cw1200_spi.c +1 -0
@@ -14,6 +14,7 @@*/ #include <linux/module.h> +#include <linux/sched.h> #include <linux/gpio.h> #include <linux/delay.h> #include <linux/spinlock.h>
drivers/net/wireless/iwlwifi/dvm/sta.c +1 -0
@@ -27,6 +27,7 @@* *****************************************************************************/ #include <linux/etherdevice.h> +#include <linux/sched.h> #include <net/mac80211.h> #include "iwl-trans.h" #include "dev.h"
drivers/net/wireless/iwlwifi/iwl-op-mode.h +1 -0
@@ -64,6 +64,7 @@#define __iwl_op_mode_h__#include <linux/debugfs.h> +#include <linux/sched.h> struct iwl_op_mode ;struct iwl_trans ;
drivers/net/wireless/iwlwifi/iwl-trans.h +1 -0
@@ -63,6 +63,7 @@#ifndef __iwl_trans_h__#define __iwl_trans_h__ +#include <linux/sched.h> #include <linux/ieee80211.h> #include <linux/mm.h> /* for page_address */ #include <linux/lockdep.h>
drivers/net/wireless/libertas_tf/cmd.c +1 -0
@@ -9,6 +9,7 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/sched.h> #include <linux/hardirq.h> #include <linux/slab.h> #include <linux/export.h>
drivers/pci/iov.c +1 -0
@@ -8,6 +8,7 @@* Address Translation Service 1.0 */ +#include <linux/sched.h> #include <linux/pci.h> #include <linux/slab.h> #include <linux/mutex.h>
drivers/pci/pci.c +1 -0
@@ -7,6 +7,7 @@* Copyright 1997 -- 2000 Martin Mares <[email protected]> */ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/delay.h> #include <linux/init.h>
drivers/platform/olpc/olpc-ec.c +1 -0
@@ -6,6 +6,7 @@* Licensed under the GPL v2 or later. */ #include <linux/completion.h> +#include <linux/sched.h> #include <linux/debugfs.h> #include <linux/spinlock.h> #include <linux/mutex.h>
drivers/ssb/driver_pcicore.c +1 -0
@@ -8,6 +8,7 @@* Licensed under the GNU/GPL. See COPYING for details. */ +#include <linux/sched.h> #include <linux/ssb/ssb.h> #include <linux/pci.h> #include <linux/export.h>
drivers/staging/lustre/lustre/llite/remote_perm.c +1 -0
@@ -44,6 +44,7 @@#define DEBUG_SUBSYSTEM S_LLITE#include <linux/module.h> +#include <linux/sched.h> #include <linux/types.h> #include <linux/version.h>
drivers/staging/lustre/lustre/obdclass/cl_lock.c +1 -0
@@ -41,6 +41,7 @@#define DEBUG_SUBSYSTEM S_CLASS#include <obd_class.h> +#include <linux/sched.h> #include <obd_support.h> #include <lustre_fid.h> #include <linux/list.h>
drivers/staging/lustre/lustre/obdclass/cl_object.c +1 -0
@@ -52,6 +52,7 @@#define DEBUG_SUBSYSTEM S_CLASS#include <linux/libcfs/libcfs.h> +#include <linux/sched.h> /* class_put_type() */ #include <obd_class.h> #include <obd_support.h>
drivers/staging/lustre/lustre/obdclass/cl_page.c +1 -0
@@ -41,6 +41,7 @@#define DEBUG_SUBSYSTEM S_CLASS#include <linux/libcfs/libcfs.h> +#include <linux/sched.h> #include <obd_class.h> #include <obd_support.h> #include <linux/list.h>
drivers/staging/lustre/lustre/osc/osc_lock.c +1 -0
@@ -43,6 +43,7 @@# include <linux/libcfs/libcfs.h> /* fid_build_reg_res_name() */ #include <lustre_fid.h> +#include <linux/sched.h> #include "osc_cl_internal.h"
drivers/staging/lustre/lustre/osc/osc_page.c +1 -0
@@ -41,6 +41,7 @@#define DEBUG_SUBSYSTEM S_OSC#include "osc_cl_internal.h" +#include <linux/sched.h> static void osc_lru_del (struct client_obd *cli,struct osc_page *opg,bool del);static void osc_lru_add (struct client_obd *cli,struct osc_page *opg);
drivers/staging/lustre/lustre/ptlrpc/client.c +1 -0
@@ -39,6 +39,7 @@#define DEBUG_SUBSYSTEM S_RPC#include <obd_support.h> +#include <linux/sched.h> #include <obd_class.h> #include <lustre_lib.h> #include <lustre_ha.h>
drivers/staging/lustre/lustre/ptlrpc/gss/gss_cli_upcall.c +1 -0
@@ -40,6 +40,7 @@#define DEBUG_SUBSYSTEM S_SEC#include <linux/init.h> +#include <linux/sched.h> #include <linux/module.h> #include <linux/slab.h> #include <linux/dcache.h>
drivers/staging/lustre/lustre/ptlrpc/gss/gss_pipefs.c +1 -0
@@ -48,6 +48,7 @@#define DEBUG_SUBSYSTEM S_SEC#include <linux/init.h> +#include <linux/sched.h> #include <linux/module.h> #include <linux/slab.h> #include <linux/dcache.h>
drivers/staging/lustre/lustre/ptlrpc/sec.c +1 -0
@@ -41,6 +41,7 @@#define DEBUG_SUBSYSTEM S_SEC#include <linux/libcfs/libcfs.h> +#include <linux/sched.h> #include <linux/crypto.h> #include <linux/key.h>
drivers/staging/lustre/lustre/ptlrpc/sec_config.c +1 -0
@@ -37,6 +37,7 @@#define DEBUG_SUBSYSTEM S_SEC#include <linux/libcfs/libcfs.h> +#include <linux/sched.h> #include <linux/crypto.h> #include <linux/key.h>
drivers/staging/lustre/lustre/ptlrpc/sec_gc.c +1 -0
@@ -41,6 +41,7 @@#define DEBUG_SUBSYSTEM S_SEC#include <linux/libcfs/libcfs.h> +#include <linux/sched.h> #include <obd_support.h> #include <obd_class.h>
drivers/usb/core/hcd.c +1 -0
@@ -23,6 +23,7 @@*/ #include <linux/bcd.h> +#include <linux/sched.h> #include <linux/module.h> #include <linux/version.h> #include <linux/kernel.h>
drivers/usb/core/urb.c +1 -0
@@ -1,3 +1,4 @@ +#include <linux/sched.h> #include <linux/module.h> #include <linux/string.h> #include <linux/bitops.h>
drivers/video/atmel_lcdfb.c +1 -0
@@ -8,6 +8,7 @@* more details. */ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/platform_device.h> #include <linux/dma-mapping.h>
fs/block_dev.c +1 -0
@@ -5,6 +5,7 @@* Copyright (C) 2001 Andrea Arcangeli <[email protected]> SuSE */ +#include <linux/sched.h> #include <linux/init.h> #include <linux/mm.h> #include <linux/fcntl.h>
fs/buffer.c +1 -0
@@ -18,6 +18,7 @@* async buffer flushing, 1999 Andrea Arcangeli <[email protected]> */ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/syscalls.h> #include <linux/fs.h>
fs/dcache.c +1 -0
@@ -14,6 +14,7 @@* the dcache entry is deleted or garbage collected. */ +#include <linux/sched.h> #include <linux/syscalls.h> #include <linux/string.h> #include <linux/mm.h>
fs/ext3/inode.c +1 -0
@@ -22,6 +22,7 @@* Assorted race fixes, rewrite of ext3_get_block() by Al Viro, 2000 */ +#include <linux/sched.h> #include <linux/highuid.h> #include <linux/quotaops.h> #include <linux/writeback.h>
fs/ext4/ext4_jbd2.c +1 -0
@@ -2,6 +2,7 @@* Interface between ext4 and JBD */ +#include <linux/sched.h> #include "ext4_jbd2.h" #include <trace/events/ext4.h>
fs/ext4/inode.c +1 -0
@@ -18,6 +18,7 @@* Assorted race fixes, rewrite of ext4_get_block() by Al Viro, 2000 */ +#include <linux/sched.h> #include <linux/fs.h> #include <linux/time.h> #include <linux/jbd2.h>
fs/ext4/mballoc.c +1 -0
@@ -22,6 +22,7 @@*/ #include "ext4_jbd2.h" +#include <linux/sched.h> #include "mballoc.h" #include <linux/log2.h> #include <linux/module.h>
fs/file_table.c +1 -0
@@ -5,6 +5,7 @@* Copyright (C) 1997 David S. Miller ([email protected]) */ +#include <linux/sched.h> #include <linux/string.h> #include <linux/slab.h> #include <linux/file.h>
fs/inode.c +1 -0
@@ -2,6 +2,7 @@* (C) 1997 Linus Torvalds * (C) 1999 Andrea Arcangeli <[email protected]> (dynamic inode allocation) */ +#include <linux/sched.h> #include <linux/export.h> #include <linux/fs.h> #include <linux/mm.h>
fs/jbd/revoke.c +1 -0
@@ -81,6 +81,7 @@*/ #ifndef __KERNEL__ +#include <linux/sched.h> #include "jfs_user.h" #else #include <linux/time.h>
fs/jbd2/revoke.c +1 -0
@@ -81,6 +81,7 @@*/ #ifndef __KERNEL__ +#include <linux/sched.h> #include "jfs_user.h" #else #include <linux/time.h>
fs/locks.c +1 -0
@@ -114,6 +114,7 @@* Stephen Rothwell <[email protected]>, June, 2000. */ +#include <linux/sched.h> #include <linux/capability.h> #include <linux/file.h> #include <linux/fdtable.h>
fs/nfs/nfs4filelayoutdev.c +1 -0
@@ -29,6 +29,7 @@*/ #include <linux/nfs_fs.h> +#include <linux/sched.h> #include <linux/vmalloc.h> #include <linux/module.h> #include <linux/sunrpc/addr.h>
fs/nfs/nfs4proc.c +1 -0
@@ -35,6 +35,7 @@* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ +#include <linux/sched.h> #include <linux/mm.h> #include <linux/delay.h> #include <linux/errno.h>
fs/nfs/nfs4state.c +1 -0
@@ -39,6 +39,7 @@*/ #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/slab.h> #include <linux/fs.h> #include <linux/nfs_fs.h>
fs/nilfs2/the_nilfs.c +1 -0
@@ -21,6 +21,7 @@* */ +#include <linux/sched.h> #include <linux/buffer_head.h> #include <linux/slab.h> #include <linux/blkdev.h>
fs/xfs/xfs_mount.c +1 -0
@@ -15,6 +15,7 @@* along with this program; if not, write the Free Software Foundation, * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ +#include <linux/sched.h> #include "xfs.h" #include "xfs_fs.h" #include "xfs_types.h"
include/asm-generic/gpio.h +1 -0
@@ -2,6 +2,7 @@#define _ASM_GENERIC_GPIO_H#include <linux/kernel.h> +#include <linux/sched.h> #include <linux/types.h> #include <linux/errno.h> #include <linux/of.h>
include/linux/buffer_head.h +1 -0
@@ -8,6 +8,7 @@#define _LINUX_BUFFER_HEAD_H#include <linux/types.h> +#include <linux/sched.h> #include <linux/fs.h> #include <linux/linkage.h> #include <linux/pagemap.h>
include/linux/clk.h +1 -0
@@ -14,6 +14,7 @@#include <linux/err.h> #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/notifier.h> struct device ;
include/linux/gpio.h +1 -0
@@ -2,6 +2,7 @@#define __LINUX_GPIO_H#include <linux/errno.h> +#include <linux/sched.h> /* see Documentation/gpio.txt */
include/linux/highmem.h +1 -0
@@ -2,6 +2,7 @@#define _LINUX_HIGHMEM_H#include <linux/fs.h> +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/bug.h> #include <linux/mm.h>
include/linux/pagemap.h +1 -0
@@ -5,6 +5,7 @@* Copyright 1995 Linus Torvalds */ #include <linux/mm.h> +#include <linux/sched.h> #include <linux/fs.h> #include <linux/list.h> #include <linux/highmem.h>
kernel/fork.c +1 -0
@@ -11,6 +11,7 @@* management can be a bitch. See 'mm/memory.c': 'copy_page_range()' */ +#include <linux/sched.h> #include <linux/slab.h> #include <linux/init.h> #include <linux/unistd.h>
kernel/freezer.c +1 -0
@@ -4,6 +4,7 @@* Originally from kernel/power/process.c */ +#include <linux/sched.h> #include <linux/interrupt.h> #include <linux/suspend.h> #include <linux/export.h>
kernel/irq/chip.c +1 -0
@@ -10,6 +10,7 @@* Detailed information is available in Documentation/DocBook/genericirq */ +#include <linux/sched.h> #include <linux/irq.h> #include <linux/msi.h> #include <linux/module.h>
kernel/nsproxy.c +1 -0
@@ -13,6 +13,7 @@* Pavel Emelianov <[email protected]> */ +#include <linux/sched.h> #include <linux/slab.h> #include <linux/export.h> #include <linux/nsproxy.h>
kernel/printk/printk.c +1 -0
@@ -17,6 +17,7 @@*/ #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/mm.h> #include <linux/tty.h> #include <linux/tty_driver.h>
kernel/sched/core.c +1 -0
@@ -26,6 +26,7 @@* Thomas Gleixner, Mike Kravetz */ +#include <linux/sched.h> #include <linux/mm.h> #include <linux/module.h> #include <linux/nmi.h>
kernel/smp.c +1 -0
@@ -9,6 +9,7 @@#include <linux/export.h> #include <linux/percpu.h> #include <linux/init.h> +#include <linux/sched.h> #include <linux/gfp.h> #include <linux/smp.h> #include <linux/cpu.h>
mm/hugetlb.c +1 -0
@@ -2,6 +2,7 @@* Generic hugetlb support. * (C) Nadia Yvette Chambers, April 2004 */ +#include <linux/sched.h> #include <linux/list.h> #include <linux/init.h> #include <linux/module.h>
mm/memory.c +1 -0
@@ -38,6 +38,7 @@* Aug/Sep 2004 Changed to four level page tables (Andi Kleen) */ +#include <linux/sched.h> #include <linux/kernel_stat.h> #include <linux/mm.h> #include <linux/hugetlb.h>
mm/mmap.c +1 -0
@@ -7,6 +7,7 @@*/ #include <linux/kernel.h> +#include <linux/sched.h> #include <linux/slab.h> #include <linux/backing-dev.h> #include <linux/mm.h>
mm/rmap.c +1 -0
@@ -42,6 +42,7 @@* pte map lock */ +#include <linux/sched.h> #include <linux/mm.h> #include <linux/pagemap.h> #include <linux/swap.h>
net/caif/cfcnfg.c +1 -0
@@ -6,6 +6,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ":%s(): " fmt, __func__ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/stddef.h> #include <linux/slab.h>
net/mac80211/driver-ops.h +1 -0
@@ -1,6 +1,7 @@#ifndef __MAC80211_DRIVER_OPS#define __MAC80211_DRIVER_OPS +#include <linux/sched.h> #include <net/mac80211.h> #include "ieee80211_i.h" #include "trace.h"
net/mac80211/key.c +1 -0
@@ -9,6 +9,7 @@* published by the Free Software Foundation. */ +#include <linux/sched.h> #include <linux/if_ether.h> #include <linux/etherdevice.h> #include <linux/list.h>
net/mac80211/main.c +1 -0
@@ -8,6 +8,7 @@* published by the Free Software Foundation. */ +#include <linux/sched.h> #include <net/mac80211.h> #include <linux/module.h> #include <linux/init.h>
net/mac80211/sta_info.c +1 -0
@@ -7,6 +7,7 @@* published by the Free Software Foundation. */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/init.h> #include <linux/etherdevice.h>
net/phonet/pep.c +1 -0
@@ -22,6 +22,7 @@* 02110-1301 USA */ +#include <linux/sched.h> #include <linux/kernel.h> #include <linux/slab.h> #include <linux/socket.h>
net/sunrpc/clnt.c +1 -0
@@ -19,6 +19,7 @@#include <linux/module.h> +#include <linux/sched.h> #include <linux/types.h> #include <linux/kallsyms.h> #include <linux/mm.h>
net/wimax/op-msg.c +1 -0
@@ -71,6 +71,7 @@* wimax_msg_alloc() * wimax_msg_send() */ +#include <linux/sched.h> #include <linux/device.h> #include <linux/slab.h> #include <net/genetlink.h>
net/wimax/op-reset.c +1 -0
@@ -28,6 +28,7 @@* disconnect and reconnect the device). */ +#include <linux/sched.h> #include <net/wimax.h> #include <net/genetlink.h> #include <linux/wimax.h>
net/wimax/op-rfkill.c +1 -0
@@ -60,6 +60,7 @@* wimax_rfkill_rm() [called by wimax_dev_add/rm()] */ +#include <linux/sched.h> #include <net/wimax.h> #include <net/genetlink.h> #include <linux/wimax.h>
net/wireless/wext-proc.c +1 -0
@@ -16,6 +16,7 @@* The content of the file is basically the content of "struct iw_statistics". */ +#include <linux/sched.h> #include <linux/module.h> #include <linux/proc_fs.h> #include <linux/seq_file.h>
sound/core/info.c +1 -0
@@ -19,6 +19,7 @@* */ +#include <linux/sched.h> #include <linux/init.h> #include <linux/time.h> #include <linux/mm.h>
virt/kvm/async_pf.c +1 -0
@@ -20,6 +20,7 @@* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. */ +#include <linux/sched.h> #include <linux/kvm_host.h> #include <linux/slab.h> #include <linux/module.h>
Signature · 1 line
1.8.3.1
PATCH 4/13 Move might_sleep and friends from kernel.h to sched.h
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
These are really related to scheduling, so they should be in sched.h Users usually will need to schedule anyways.
The advantage of having them there is that we can access some of the scheduler inlines to make their fast path more efficient. This will come in a followon patch.
--- include/linux/kernel.h | 35 ----------------------------------- include/linux/sched.h | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 35 deletions(-)
include/linux/kernel.h +0 -35
@@ -141,35 +141,6 @@ struct completion;struct pt_regs ;struct user ; -#ifdef CONFIG_PREEMPT_VOLUNTARY -extern int _cond_resched (void ); -# define might_resched () _cond_resched() -#else -# define might_resched () do { } while (0) -#endif - -#ifdef CONFIG_DEBUG_ATOMIC_SLEEP -void __might_sleep (const char *file,int line,int preempt_offset); -/** -* might_sleep - annotation for functions that can sleep -* -* this macro will print a stack trace if it is executed in an atomic -* context (spinlock, irq-handler, ...). -* -* This is a useful debugging help to be able to catch problems early and not -* be bitten later when the calling function happens to sleep when it is not -* supposed to. -*/ -# define might_sleep () \ - do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0) -#else -static inline void __might_sleep (const char *file,int line, -int preempt_offset) { } -# define might_sleep () do { might_resched(); } while (0) -#endif - -#define might_sleep_if (cond) do { if (cond) might_sleep(); } while (0) -/* * abs() handles unsigned and signed longs, ints, shorts and chars. For all * input types abs() returns a signed long.
@@ -193,12 +164,6 @@ extern int _cond_resched(void); (__x <0 ) ? -__x : __x; \ }) -#if defined(CONFIG_PROVE_LOCKING ) || defined(CONFIG_DEBUG_ATOMIC_SLEEP ) -void might_fault (void ); -#else -static inline void might_fault (void ) { } -#endif -extern struct atomic_notifier_head panic_notifier_list;extern long (*panic_blink)(int state);__printf (1 ,2 )
include/linux/sched.h +38 -0
@@ -2433,6 +2433,35 @@ extern int __cond_resched_softirq(void);__cond_resched_softirq (); \ }) +#ifdef CONFIG_PREEMPT_VOLUNTARY +extern int _cond_resched (void ); +# define might_resched () _cond_resched() +#else +# define might_resched () do { } while (0) +#endif + +#ifdef CONFIG_DEBUG_ATOMIC_SLEEP +void __might_sleep (const char *file,int line,int preempt_offset); +/** +* might_sleep - annotation for functions that can sleep +* +* this macro will print a stack trace if it is executed in an atomic +* context (spinlock, irq-handler, ...). +* +* This is a useful debugging help to be able to catch problems early and not +* be bitten later when the calling function happens to sleep when it is not +* supposed to. +*/ +# define might_sleep () \ + do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0) +#else +static inline void __might_sleep (const char *file,int line, +int preempt_offset) { } +# define might_sleep () do { might_resched(); } while (0) +#endif + +#define might_sleep_if (cond) do { if (cond) might_sleep(); } while (0) +static inline void cond_resched_rcu (void ) {#if defined(CONFIG_DEBUG_ATOMIC_SLEEP ) || !defined(CONFIG_PREEMPT_RCU )
@@ -2442,6 +2471,15 @@ static inline void cond_resched_rcu(void)#endif } +#ifdef CONFIG_PROVE_LOCKING +void might_fault (void ); +#else +static inline void might_fault (void ) +{ +might_sleep (); +} +#endif +/* * Does a critical section need to be broken due to another * task waiting?: (technically does not depend on CONFIG_PREEMPT,
Signature · 1 line
1.8.3.1
PATCH 5/13 sched: mark should_resched() __always_inline
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
At least gcc 4.6 and some earlier ones does not inline this function. Since it's small and on relatively hot paths force inline it.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -3767,7 +3767,7 @@ SYSCALL_DEFINE0(sched_yield)return 0 ; } -static inline int should_resched (void ) +static __always_inline int should_resched(void) {return need_resched () && !(preempt_count () &PREEMPT_ACTIVE ); }
Signature · 1 line
1.8.3.1
PATCH 6/13 x86: Add 32bit versions of SAVE_ALL/RESTORE_ALL to calling.h
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add 32bit versions of SAVE/RESTORE_ALL to calling.h. Needed for the following patches. These are much simplified over both 64bit and the 32bit entry* version, just saving and restoring all registers without anything fancy.
These are different from the entry_32.S versions in not changing the segment registers or cld. So only suitable for in kernel use, not when transitioning from or to user space. The resulting stack frame is not a standard pt_regs frame. The main use case is calling C code from assembler when all the registers need to be preserved.
--- arch/x86/include/asm/calling.h | 49 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
arch/x86/include/asm/calling.h +49 -0
@@ -48,6 +48,8 @@ For 32-bit we have the following conventions - kernel is built with#include <asm/dwarf2.h> +#ifdef CONFIG_X86_64 +/* * 64-bit system call stack frame layout defines and helpers, * for assembly code:
@@ -192,3 +194,50 @@ For 32-bit we have the following conventions - kernel is built with .macro icebp .byte 0xf1 .endm + +#else + +/* +* For 32bit only simplified versions of SAVE_ALL/RESTORE_ALL. These +* are different from the entry_32.S versions in not changing the segment +* registers. So only suitable for in kernel use, not when transitioning +* from or to user space. The resulting stack frame is not a standard +* pt_regs frame. The main use case is calling C code from assembler +* when all the registers need to be preserved. +*/ + + .macro SAVE_ALL + pushl_cfi %eax + CFI_REL_OFFSET eax, 0 + pushl_cfi %ebp + CFI_REL_OFFSET ebp, 0 + pushl_cfi %edi + CFI_REL_OFFSET edi, 0 + pushl_cfi %esi + CFI_REL_OFFSET esi, 0 + pushl_cfi %edx + CFI_REL_OFFSET edx, 0 + pushl_cfi %ecx + CFI_REL_OFFSET ecx, 0 + pushl_cfi %ebx + CFI_REL_OFFSET ebx, 0 + .endm + + .macro RESTORE_ALL + popl_cfi %ebx + CFI_RESTORE ebx + popl_cfi %ecx + CFI_RESTORE ecx + popl_cfi %edx + CFI_RESTORE edx + popl_cfi %esi + CFI_RESTORE esi + popl_cfi %edi + CFI_RESTORE edi + popl_cfi %ebp + CFI_RESTORE ebp + popl_cfi %eax + CFI_RESTORE eax + .endm + +#endif
Signature · 1 line
1.8.3.1
PATCH 7/13 Add might_fault_debug_only()
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add a might_fault_debug_only() that only does something in the PROVE_LOCKING case, but does not cond_resched for PREEMPT_VOLUNTARY. This is for cases when the cond_resched is done elsewhere
--- include/linux/sched.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/sched.h +2 -0
@@ -2473,11 +2473,13 @@ static inline void cond_resched_rcu(void)#ifdef CONFIG_PROVE_LOCKING void might_fault (void ); +#define might_fault_debug_only () might_fault()#else static inline void might_fault (void ) {might_sleep (); } +#define might_fault_debug_only () do {} while(0)#endif /*
Signature · 1 line
1.8.3.1
Re: [PATCH 07/13] Add might_fault_debug_only()
Quoted 16 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Add a
might_fault_debug_only()that only does something in thePROVE_LOCKINGcase, but does notcond_reschedforPREEMPT_VOLUNTARY. This is for cases when thecond_reschedis done elsewhereSigned-off-by Andi Kleen <[email protected]>--- include/linux/sched.h | 2 ++ 1 file changed, 2 insertions(+)include/linux/sched.h +0
-0
include/linux/sched.h static inline void cond_resched_rcu(void) · line 2482
Quoted 10 lines
include/linux/sched.h +2
-0@@ -2473,11 +2473,13 @@ static inline void cond_resched_rcu(void)#ifdef CONFIG_PROVE_LOCKING void might_fault (void ); +#define might_fault_debug_only () might_fault()#else static inline void might_fault (void ) {might_sleep (); } +#define might_fault_debug_only () do {} while(0)
Hi Andy, this is against which kernel version?
In 3.11-rc3 I see:
#if defined(CONFIG_PROVE_LOCKING) || defined(CONFIG_DEBUG_ATOMIC_SLEEP) void might_fault(void);
#else
static inline void might_fault(void) { }
#endif
So it's not clear to me how it's different from your might_fault_debug_only ..
Quoted 12 lines
include/linux/sched.h +0
-0#endif /* Signature · 7 lines
1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
PATCH 8/13 x86: Move cond_resched into the out of line put_user code
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
CONFIG_PREEMPT_VOLUNTARY kernels always do a cond_resched in put_user(). Currently this is done in the caller and in a inefficient way (multiple function calls to decide to do nothing). Move the reschedule check into the low level functions instead, where it can be merged cheaply with the address limit check.
For the DEBUG_SLEEP case we still do the call in the caller.
--- arch/x86/include/asm/uaccess.h | 2 +- arch/x86/lib/putuser.S | 12 +++++++++++- arch/x86/lib/user-common.h | 12 ++++++++++++ 3 files changed, 24 insertions(+), 2 deletions(-) create mode 100644 arch/x86/lib/user-common.h
arch/x86/include/asm/uaccess.h +1 -1
@@ -246,7 +246,7 @@ extern void __put_user_8(void);int __ret_pu; \ __typeof__(*(ptr)) __pu_val; \__chk_user_ptr (ptr); \ -might_fault (); \ +might_fault_debug_only (); \ __pu_val = x; \switch (sizeof (*(ptr))) { \case 1 : \
arch/x86/lib/putuser.S +11 -1
@@ -16,6 +16,8 @@ #include <asm/errno.h> #include <asm/asm.h> #include <asm/smap.h> +#include <asm/calling.h> +#include "user-common.h" /*
@@ -31,7 +33,7 @@ */ #define ENTER CFI_STARTPROC ; \ - GET_THREAD_INFO(%_ASM_BX) + GET_THREAD_AND_SCHEDULE %_ASM_BX #define EXIT ASM_CLAC ; \ ret ; \ CFI_ENDPROC
@@ -99,3 +101,11 @@ END(bad_put_user) #ifdef CONFIG_X86_32 _ASM_EXTABLE(5b,bad_put_user) #endif + +ENTRY(user_schedule) + CFI_STARTPROC + SAVE_ALL + call _cond_resched + RESTORE_ALL + ret + CFI_ENDPROC
arch/x86/lib/user-common.h new file +12 -0
@@ -0,0 +1,12 @@ + .macro GET_THREAD_AND_SCHEDULE reg + GET_THREAD_INFO(\reg) +#ifdef CONFIG_PREEMPT_VOLUNTARY + testl $_TIF_NEED_RESCHED,TI_flags(\reg) + jnz 1f +2: + .section .fixup,"ax" +1: call user_schedule + jmp 2b + .previous +#endif + .endm
Signature · 1 line
1.8.3.1
PATCH 9/13 x86: Move cond_resched into the out of line get_user code
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
CONFIG_PREEMPT_VOLUNTARY kernels always do a cond_resched in get_user(). Currently this is done in the caller and in a inefficient way (multiple function calls to decide to do nothing). Move the reschedule check into the low level functions instead, where it can be merged cheaply with the address limit check.
For the DEBUG_SLEEP case we still do the call in the caller.
--- arch/x86/include/asm/uaccess.h | 3 +-- arch/x86/lib/getuser.S | 9 +++++---- 2 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/uaccess.h +1 -2
@@ -165,7 +165,7 @@ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL))int __ret_gu; \ register __inttype(*(ptr)) __val_gu asm("%edx"); \__chk_user_ptr (ptr); \ -might_fault (); \ +might_fault_debug_only (); \ asmvolatile ("call __get_user_%P3" \ :"=a" (__ret_gu),"=r" (__val_gu) \ :"0" (ptr),"i" (sizeof (*(ptr)))); \
@@ -541,4 +541,3 @@ extern struct movsl_mask {#endif #endif /* _ASM_X86_UACCESS_H */ -
arch/x86/lib/getuser.S +5 -4
@@ -33,11 +33,12 @@ #include <asm/thread_info.h> #include <asm/asm.h> #include <asm/smap.h> +#include "user-common.h" .text ENTRY(__get_user_1) CFI_STARTPROC - GET_THREAD_INFO(%_ASM_DX) + GET_THREAD_AND_SCHEDULE %_ASM_DX cmp TI_addr_limit(%_ASM_DX),%_ASM_AX jae bad_get_user ASM_STAC
@@ -52,7 +53,7 @@ ENTRY(__get_user_2) CFI_STARTPROC add $1,%_ASM_AX jc bad_get_user - GET_THREAD_INFO(%_ASM_DX) + GET_THREAD_AND_SCHEDULE %_ASM_DX cmp TI_addr_limit(%_ASM_DX),%_ASM_AX jae bad_get_user ASM_STAC
@@ -67,7 +68,7 @@ ENTRY(__get_user_4) CFI_STARTPROC add $3,%_ASM_AX jc bad_get_user - GET_THREAD_INFO(%_ASM_DX) + GET_THREAD_AND_SCHEDULE %_ASM_DX cmp TI_addr_limit(%_ASM_DX),%_ASM_AX jae bad_get_user ASM_STAC
@@ -83,7 +84,7 @@ ENTRY(__get_user_8) #ifdef CONFIG_X86_64 add $7,%_ASM_AX jc bad_get_user - GET_THREAD_INFO(%_ASM_DX) + GET_THREAD_AND_SCHEDULE %_ASM_DX cmp TI_addr_limit(%_ASM_DX),%_ASM_AX jae bad_get_user ASM_STAC
Signature · 1 line
1.8.3.1
PATCH 10/13 x86: Move cond resched for copy_{from,to}_user into low level code 64bit
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Move the cond_resched() check for CONFIG_PREEMPT_VOLUNTARY into the low level copy_*_user code. This avoids some code bloat and makes check much more efficient by avoiding unnecessary function calls.
This is currently only for the non __ variants.
For the sleep debug case the call is still done in the caller.
I did not do this for copy_in_user() or the nocache variants because there's no obvious place to put the check, and those calls are comparatively rare.
--- arch/x86/include/asm/uaccess_64.h | 4 ++-- arch/x86/lib/copy_user_64.S | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/include/asm/uaccess_64.h +2 -2
@@ -58,7 +58,7 @@ static inline unsigned long __must_check copy_from_user(void *to, {int sz =__compiletime_object_size (to); -might_fault (); +might_fault_debug_only ();if (likely (sz ==-1 || sz >= n)) n =_copy_from_user (to, from, n);#ifdef CONFIG_DEBUG_VM
@@ -71,7 +71,7 @@ static inline unsigned long __must_check copy_from_user(void *to,static __always_inline __must_check int copy_to_user(void __user *dst, const void *src, unsigned size) { -might_fault (); +might_fault_debug_only ();return _copy_to_user (dst, src, size); }
arch/x86/lib/copy_user_64.S +3 -2
@@ -18,6 +18,7 @@ #include <asm/alternative-asm.h> #include <asm/asm.h> #include <asm/smap.h> +#include "user-common.h" /* * By placing feature2 after feature1 in altinstructions section, we logically
@@ -73,7 +74,7 @@ /* Standard copy_to_user with segment limit checking */ ENTRY(_copy_to_user) CFI_STARTPROC - GET_THREAD_INFO(%rax) + GET_THREAD_AND_SCHEDULE %rax movq %rdi,%rcx addq %rdx,%rcx jc bad_to_user
@@ -88,7 +89,7 @@ ENDPROC(_copy_to_user) /* Standard copy_from_user with segment limit checking */ ENTRY(_copy_from_user) CFI_STARTPROC - GET_THREAD_INFO(%rax) + GET_THREAD_AND_SCHEDULE %rax movq %rsi,%rcx addq %rdx,%rcx jc bad_from_user
Signature · 1 line
1.8.3.1
Re: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
Quoted 5 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Move the
cond_resched()check forCONFIG_PREEMPT_VOLUNTARYinto the low level copy_*_user code. This avoids some code bloat and makes check much more efficient by avoiding unnecessary function calls.
May I suggest going one step further, and just removing the cond_resched() _entirely_, leaving just the debug test?
There really is zero reason for doing a cond_resched() for user accesses. If they take a page fault, then yes, by all means do that (and maybe we should add one to the page fault trap if we don't have it already), but without a page fault they really aren't that expensive.
We do many more expensive things without any cond_resched(), and doing that cond_resched() really doesn't make much sense *unless* there's a big expensive loop involved.
Most of this series looks fine, but I really think that we could/should just take that extra step, and say "no, user accesses don't imply that we need to check for scheduling".
Linus
Re: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
Quoted 3 lines
Most of this series looks fine, but I really think that we could/should just take that extra step, and say "no, user accesses don't imply that we need to check for scheduling".
Hmm. I can do that, but wouldn't that make CONFIG_PREEMPT_VOLUNTARY mostly equivalent to CONFIG_PREEMPT_NONE?
Need to check how many other reschedule tests are left then for VOLUNTARY.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
Quoted 3 lines · Andi Kleen
Hmm. I can do that, but wouldn't that make
CONFIG_PREEMPT_VOLUNTARYmostly equivalent toCONFIG_PREEMPT_NONE?
According the the Kconfig help, PREEMPT_VOLUNTARY is about the *explicit* preemption points. And we do have a lot of them in "might_sleep()".
And personally, I think it makes a *lot* more sense to have a "might_sleep()" in the MM allocators than it does to have it in copy_from_user().
But yes, it's obviously a gray area on exactly where you'd want to put them. But for a "get_user()" that can literally be just a few instructions? Why would we make that a preemption point? Or even a copy of few tens of bytes?
I'd *much* rather have preemption points in places that actually loop, and that have real work associated with them.
Now, the *debug* logic is entirely different, of course. Maybe the problem is that we have mixed up the two so badly, and we have "might_sleep()" that implies more of a debug issue than a preemption issue, and then people add those because they want the debug coverage (and then you *absolutely* want it even for a single-byte user mode access). And then because the concept is tied together with preemption, we end up doing preemption even for that single-byte access despite the fact that it makes no sense what-so-ever.
So I think your patches are fine, but I do think we should take a deeper look at this.
Linus
Borislav Petkov
maintainerRe: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
Quoted 8 lines · Linus Torvalds
Now, the *debug* logic is entirely different, of course. Maybe the problem is that we have mixed up the two so badly, and we have "
might_sleep()" that implies more of a debug issue than a preemption issue, and then people add those because they want the debug coverage (and then you *absolutely* want it even for a single-byte user mode access). And then because the concept is tied together with preemption, we end up doing preemption even for that single-byte access despite the fact that it makes no sense what-so-ever.
Sounds like the debug aspect and the preemption point addition need to be sorf-of split into two different functions/macros and each used separately.
Something like keep the current might_sleep and have debug_sleep or similar which does only __might_sleep without the resched...
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Re: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
Quoted 7 lines · Borislav Petkov
Sounds like the debug aspect and the preemption point addition need to be sorf-of split into two different functions/macros and each used separately.
Something like keep the current
might_sleepand havedebug_sleepor similar which does only__might_sleepwithout the resched...
I would argue for using "might_sleep" for the debug variant. Before reading this thread I wasn't even aware of the non-debug aspect. After all, might_sleep naturally reads like some assertion. "might_preempt" for the non-debug version? "cond_preempt"?
Jörn
--
It is the mark of an educated mind to be able to entertain a thought without accepting it.
-- Aristotle
Re: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
In reply to Linus TorvaldsQuoted 10 lines
Quoted 2 lines
Hmm. I can do that, but wouldn't that make
CONFIG_PREEMPT_VOLUNTARYmostly equivalent toCONFIG_PREEMPT_NONE?According the the
Kconfighelp,PREEMPT_VOLUNTARYis about the *explicit* preemption points. And we do have a lot of them in "might_sleep()".And personally, I think it makes a *lot* more sense to have a "
might_sleep()" in the MM allocators than it does to have it incopy_from_user().
AFAIK, MM allocation already does that.
struct page *__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, nodemask_t *nodemask) {
(snip)
might_sleep_if(gfp_mask & __GFP_WAIT);
btw, Sorry for the very late response. I haven't noticed this thread.
Re: [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit
In reply to Linus TorvaldsQuoted 20 lines · Linus Torvalds
Quoted 5 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Move the
cond_resched()check forCONFIG_PREEMPT_VOLUNTARYinto the low level copy_*_user code. This avoids some code bloat and makes check much more efficient by avoiding unnecessary function calls.May I suggest going one step further, and just removing the
cond_resched()_entirely_, leaving just the debug test?There really is zero reason for doing a
cond_resched()for user accesses. If they take a page fault, then yes, by all means do that (and maybe we should add one to the page fault trap if we don't have it already), but without a page fault they really aren't that expensive.We do many more expensive things without any
cond_resched(), and doing thatcond_resched()really doesn't make much sense *unless* there's a big expensive loop involved.
Quoted 5 lines
Most of this series looks fine, but I really think that we could/should just take that extra step, and say "no, user accesses don't imply that we need to check for scheduling".
Linus
In fact we are doing exactly this since 3.11-rc1.
Signature · 1 line
MST
PATCH 11/13 sched: Inline the need_resched test into the caller for _cond_resched
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
_cond_resched is very common in kernel calls, e.g. it's used in every user access. Usually it does at least two explicit calls just to decide to do
nothing: _cond_resched and should_resched(). Inline a need_resched() into the caller to avoid these calls in the common case of no reschedule being needed.
Previously this would have been very expensive in terms of binary size because there were a lot of inlined cond_resched()s in copy_*_user() and put/get_user(). But with the newest changes to x86 uaccess.h these not inlined anymore, so we can use a slightly bigger, but much faster fast path version.
--- include/linux/sched.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/sched.h +1 -1
@@ -2435,7 +2435,7 @@ extern int __cond_resched_softirq(void);#ifdef CONFIG_PREEMPT_VOLUNTARY extern int _cond_resched (void ); -# define might_resched () _cond_resched() +# define might_resched () (need_resched() ? _cond_resched() : 0)#else # define might_resched () do { } while (0)#endif
Signature · 1 line
1.8.3.1
PATCH 12/13 x86: move __copy_*_nocache might fault check out of line
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Can as well do the normal conditional resched check out of line. This saves one function call.
--- arch/x86/include/asm/uaccess_64.h | 6 ++++-- arch/x86/lib/copy_user_nocache_64.S | 7 +++++++ 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/include/asm/uaccess_64.h +4 -2
@@ -243,12 +243,14 @@ __copy_to_user_inatomic(void __user *dst, const void *src, unsigned size)extern long __copy_user_nocache (void *dst,const void __user *src,unsigned size,int zerorest); +extern long __copy_user_nocache_might_fault (void *dst,const void __user *src, +unsigned size,int zerorest);static inline int __copy_from_user_nocache (void *dst,const void __user *src,unsigned size) { -might_fault (); -return __copy_user_nocache (dst, src, size,1 ); +might_fault_debug_only (); +return __copy_user_nocache_might_fault (dst, src, size,1 ); } static inline int
arch/x86/lib/copy_user_nocache_64.S +7 -0
@@ -16,6 +16,7 @@ #include <asm/thread_info.h> #include <asm/asm.h> #include <asm/smap.h> +#include "user-common.h" .macro ALIGN_DESTINATION #ifdef FIX_ALIGNMENT
@@ -43,6 +44,12 @@ #endif .endm +ENTRY(__copy_user_nocache_might_fault) + CFI_STARTPROC + GET_THREAD_AND_SCHEDULE %rax + CFI_ENDPROC + /* fall through */ + /* * copy_user_nocache - Uncached memory copy with exception handling * This will force destination/source out of cache for more performance.
Signature · 1 line
1.8.3.1
PATCH 13/13 x86: drop cond rescheds from __copy_{from,to}_user
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
The __copy_* variants are right now more expensive than the non __ copy*user in CONFIG_PREEMPT_VOLUNTARY because they have a additional function call to might_fault().
Since they are usually used in a row with other functions, which also schedule or only in the thin compat layers and also __get/__put_user do not do explicit reschedule check drop them here for the non debug case.
Normal non __ copy*user will still schedule of course
--- arch/x86/include/asm/uaccess_32.h | 6 +++--- arch/x86/include/asm/uaccess_64.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/uaccess_32.h +3 -3
@@ -81,7 +81,7 @@ __copy_to_user_inatomic(void __user *to, const void *from, unsigned long n)static __always_inline unsigned long __must_check __copy_to_user (void __user *to,const void *from,unsigned long n) { -might_fault (); +might_fault_debug_only ();return __copy_to_user_inatomic (to, from, n); }
@@ -136,7 +136,7 @@ __copy_from_user_inatomic(void *to, const void __user *from, unsigned long n)static __always_inline unsigned long __copy_from_user (void *to,const void __user *from,unsigned long n) { -might_fault (); +might_fault_debug_only ();if (__builtin_constant_p (n)) {unsigned long ret;
@@ -158,7 +158,7 @@ __copy_from_user(void *to, const void __user *from, unsigned long n)static __always_inline unsigned long __copy_from_user_nocache (void *to,const void __user *from,unsigned long n) { -might_fault (); +might_fault_debug_only ();if (__builtin_constant_p (n)) {unsigned long ret;
arch/x86/include/asm/uaccess_64.h +2 -2
@@ -122,7 +122,7 @@ int __copy_from_user_nocheck(void *dst, const void __user *src, unsigned size)static __always_inline __must_check int __copy_from_user(void *dst, const void __user *src, unsigned size) { -might_fault (); +might_fault_debug_only ();return __copy_from_user_nocheck (dst, src, size); }
@@ -172,7 +172,7 @@ int __copy_to_user_nocheck(void __user *dst, const void *src, unsigned size)static __always_inline __must_check int __copy_to_user(void __user *dst, const void *src, unsigned size) { -might_fault (); +might_fault_debug_only ();return __copy_to_user_nocheck (dst, src, size); }
Signature · 1 line
1.8.3.1
H. Peter Anvin
reviewerQuoted 9 lines · Andi Kleen
This patch kit is an attempt to get us back to sane code, mostly by doing proper inlining and doing sleep checks in the right place. Unfortunately I had to add one tree sweep to avoid an nasty include loop.
It costs a bit of text space, but I think it's worth it (if only to keep my blood pressure down while reading ftrace logs...)
Looks nice at first glance.
Now, here is a bigger question: shouldn't we be deprecating/getting rid of PREEMPT_VOUNTARY in favor of PREEMPT?
-hpa
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Andi Kleen
This patch kit is an attempt to get us back to sane code, mostly by doing proper inlining and doing sleep checks in the right place. Unfortunately I had to add one tree sweep to avoid an nasty include loop.
Quoted 8 lines
Quoted 3 lines
It costs a bit of text space, but I think it's worth it (if only to keep my blood pressure down while reading ftrace logs...)
Looks nice at first glance.
Now, here is a bigger question: shouldn't we be deprecating/getting rid of
PREEMPT_VOUNTARYin favor of PREEMPT?
I sure hope not, PREEMPT munches throughput. If you need PREEMPT, seems to me what you _really_ need is PREEMPT_RT (the real deal), so eventually depreciating PREEMPT makes more sense to me.
-Mike
H. Peter Anvin
reviewerQuoted 8 lines · Mike Galbraith
Quoted 3 lines
Now, here is a bigger question: shouldn't we be deprecating/getting rid of
PREEMPT_VOUNTARYin favor of PREEMPT?I sure hope not, PREEMPT munches throughput. If you need PREEMPT, seems to me what you _really_ need is
PREEMPT_RT(the real deal), so eventually depreciating PREEMPT makes more sense to me.
Do you have any quantification of "munches throughput?" It seems odd that it would be worse than polling for preempt all over the kernel, but perhaps the additional locking is what costs.
-hpa
Quoted 4 lines · H. Peter Anvin
Do you have any quantification of "munches throughput?" It seems odd that it would be worse than polling for preempt all over the kernel, but perhaps the additional locking is what costs.
Actually, the big thing for true preemption is not so much the preempt count itself, but the fact that when the preempt count goes back to zero we have that "check if we should have been preempted" thing.
And in particular, the conditional function call that goes along with it.
The thing is, even if that is almost never taken, just the fact that there is a conditional function call very often makes code generation *much* worse. A function that is a leaf function with no stack frame with no preemption often turns into a non-leaf function with stack frames when you enable preemption, just because it had a RCU read region which disabled preemption.
It's similar to the kind of code generation issue that Andi's patches are trying to work on.
Andi did the "test and jump to a different section to call the scheduler with registers saved" as an assembly stub in one of his patches in this series exactly to avoid the cost of this for the might_sleep() case, and generated that GET_THREAD_AND_SCHEDULE asm macro for it. But look at that asm macro, and compare it to "preempt_check_resched()"..
I have often wanted to have access to that kind of thing from C code. It's not unusual. Think lock failure paths, not Tom Jones.
Linus
H. Peter Anvin
reviewerQuoted 29 lines · Linus Torvalds
Quoted 4 lines · H. Peter Anvin
Do you have any quantification of "munches throughput?" It seems odd that it would be worse than polling for preempt all over the kernel, but perhaps the additional locking is what costs.
Actually, the big thing for true preemption is not so much the preempt count itself, but the fact that when the preempt count goes back to zero we have that "check if we should have been preempted" thing.
And in particular, the conditional function call that goes along with it.
The thing is, even if that is almost never taken, just the fact that there is a conditional function call very often makes code generation *much* worse. A function that is a leaf function with no stack frame with no preemption often turns into a non-leaf function with stackcheck frames when you enable preemption, just because it had a RCU read region which disabled preemption.
It's similar to the kind of code generation issue that Andi's patches are trying to work on.
Andi did the "test and jump to a different section to call the scheduler with registers saved" as an assembly stub in one of his patches in this series exactly to avoid the cost of this for the
might_sleep()case, and generated thatGET_THREAD_AND_SCHEDULEasm macro for it. But look at that asm macro, and compare it to "preempt_check_resched()"..
Quoted 3 lines
I have often wanted to have access to that kind of thing from C code. It's not unusual. Think lock failure paths, not Tom Jones.
Hmm... if that is really the big issue then I'm wondering if preempt_enable() &c shouldn't be rewritten in assembly... if nothing else to get the outbound call out of view of the C compiler; it could even be turned into an exception instruction.
There are a few other things that one have to wonder about: the preempt_count is currently located in the thread_info structure, but since by definition we can't switch a thread that is preemption-locked it should work in a percpu variable as well.
We could then play a really ugly stunt by marking NEED_RESCHED by adding 0x7fffffff to the counter. Then the whole sequence becomes something like:
subl $1,%fs:preempt_count
jno 1f
call __naked_preempt_schedule /* Or a trap */
1:
For architectures with conditional traps the trapping option becomes even more attractive.
-hpa
Quoted 7 lines · H. Peter Anvin
We could then play a really ugly stunt by marking
NEED_RESCHEDby adding0x7fffffffto the counter. Then the whole sequence becomes something like:subl $1,%fs:preempt_count
jno 1f
call__naked_preempt_schedule/* Or a trap */
This is indeed one of the few cases where we probably *could* use trapv or something like that in theory, but those instructions tend to be slow enough that even if you don't take the trap, you'd be better off just testing by hand.
However, it's worse than you think. Preempt count is per-thread, not per-cpu. So to access preempt-count, we currently have to look up thread_info (which is per-cpu or stack-based).
I'd *like* to make preempt-count be per-cpu, and then copy it at thread switch time, and it's been discussed. But as things are now, preemption enable is quite expensive, and looks something like
movq %gs:kernel_stack,%rdx #, pfo_ret__
subl $1, -8124(%rdx) #, ti_22->preempt_count
movq %gs:kernel_stack,%rdx #, pfo_ret__
movq -8136(%rdx), %rdx # MEM[(const long unsigned int
*)ti_27 + 16B], D.
andl $8, %edx #, D.34545
jne .L139 #,and that's actually the *good* case (ie not counting any extra costs of turning leaf functions into non-leaf ones).
That "kernel_stack" thing is actually getting the thread_info pointer, and it doesn't get cached because gcc thinks the preempt_count value might alias. Sad, sad, sad. We actually used to do better back when we used actual tricks with the stack registers and used a const inline asm to let gcc know it could re-use the value etc.
It would be *lovely* if we
(a) made preempt-count per-cpu and just copied it at thread-switch
(b) made the NEED_RESCHED bit be part of preempt-count (rather than
thread flags) and just made it the high bit
adn then maybe we could just do
subl $1, %fs:preempt_count
js .L139
with the actual schedule call being done as an
asm volatile("call user_schedule": : :"memory");that Andi introduced that doesn't pollute the register space. Note that you still want the *test* to be done in C code, because together with "unlikely()" you'd likely do pretty close to optimal code generation, and hiding the decrement and test and conditional jump in asm you wouldn't get the proper instruction scheduling and branch following that gcc does.
I dunno. It looks like a fair amount of effort. But as things are now, the code generation difference between PREEMPT_NONE and PREEMPT is actually fairly noticeable. And PREEMPT_VOLUNTARY - which is supposed to be almost as cheap as PREEMPT_NONE - has lots of bad cases too, as Andi noticed.
Linus
H. Peter Anvin
reviewerRight... I mentioned the need to move thread count into percpu and the other restructuring... all of that seems essential for this not to suck.
Quoted 69 lines · Linus Torvalds
Quoted 2 lines · H. Peter Anvin
We could then play a really ugly stunt by marking
NEED_RESCHEDbyadding
Quoted 1 line
0x7fffffffto the counter. Then the whole sequence becomes somethinglike:
Quoted 4 lines
subl $1,%fs:preempt_count
jno 1f
call__naked_preempt_schedule/* Or a trap */This is indeed one of the few cases where we probably *could* use trapv or something like that in theory, but those instructions tend to be slow enough that even if you don't take the trap, you'd be better off just testing by hand.
However, it's worse than you think. Preempt count is per-thread, not per-cpu. So to access preempt-count, we currently have to look up
thread_info(which is per-cpu or stack-based).I'd *like* to make preempt-count be per-cpu, and then copy it at thread switch time, and it's been discussed. But as things are now, preemption enable is quite expensive, and looks something like
movq %gs:kernel_stack,%rdx #,
pfo_ret__
subl $1, -8124(%rdx) #,ti_22->preempt_count
movq %gs:kernel_stack,%rdx #,pfo_ret__
movq -8136(%rdx), %rdx # MEM[(const long unsigned int
*)ti_27 + 16B], D.andl $8, %edx #, D.34545 jne .L139 #,and that's actually the *good* case (ie not counting any extra costs of turning leaf functions into non-leaf ones).
That "
kernel_stack" thing is actually getting thethread_infopointer, and it doesn't get cached because gcc thinks thepreempt_countvalue might alias. Sad, sad, sad. We actually used to do better back when we used actual tricks with the stack registers and used a const inline asm to let gcc know it could re-use the value etc.It would be *lovely* if we
(a) made preempt-count per-cpu and just copied it at thread-switch
(b) made theNEED_RESCHEDbit be part of preempt-count (rather than thread flags) and just made it the high bitadn then maybe we could just do
subl $1, %fs:preempt_count
js .L139with the actual schedule call being done as an
asm volatile("call user_schedule": : :"memory");that Andi introduced that doesn't pollute the register space. Note that you still want the *test* to be done in C code, because together with "
unlikely()" you'd likely do pretty close to optimal code generation, and hiding the decrement and test and conditional jump in asm you wouldn't get the proper instruction scheduling and branch following that gcc does.I dunno. It looks like a fair amount of effort. But as things are now, the code generation difference between
PREEMPT_NONEand PREEMPT is actually fairly noticeable. AndPREEMPT_VOLUNTARY- which is supposed to be almost as cheap asPREEMPT_NONE- has lots of bad cases too, as Andi noticed.Linus
Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
H. Peter Anvin
reviewerQuoted 3 lines · Linus Torvalds
That "
kernel_stack" thing is actually getting thethread_infopointer, and it doesn't get cached because gcc thinks thepreempt_countvalue might alias.
This is just plain braindamaged. Somewhere on my list of things is to merge thread_info and task_struct and have a pointer to the unified structure.
-hpa
H. Peter Anvin
reviewerQuoted 5 lines · Linus Torvalds
Note that you still want the *test* to be done in C code, because together with "
unlikely()" you'd likely do pretty close to optimal code generation, and hiding the decrement and test and conditional jump in asm you wouldn't get the proper instruction scheduling and branch following that gcc does.
How much instruction scheduling can gcc actually do given that there is a barrier involved? I guess some on the RISC architectures which need load/subtract/store...
-hpa
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines · Mike Galbraith
Quoted 3 lines
Now, here is a bigger question: shouldn't we be deprecating/getting rid of
PREEMPT_VOUNTARYin favor of PREEMPT?
Quoted 8 lines
Quoted 4 lines
I sure hope not, PREEMPT munches throughput. If you need PREEMPT, seems to me what you _really_ need is
PREEMPT_RT(the real deal), so eventually depreciating PREEMPT makes more sense to me.Do you have any quantification of "munches throughput?" It seems odd that it would be worse than polling for preempt all over the kernel, but perhaps the additional locking is what costs.
I hadn't compared in ages, so made some fresh samples.
Q6600 3.11-rc4
vmark
voluntary 169808 155826 154741 1.000 preempt 149354 124016 128436 .836
That should be ~worst case, it hates preemption.
tbench 8
voluntary 1027.96 1028.76 1044.60 1.000
preempt 929.06 935.01 928.64 .900
hackbench -l 10000
voluntary 23.146 23.124 23.230 1.000 preempt 25.065 24.633 24.789 1.071
kbuild vmlinux
voluntary 3m44.842s 3m42.975s 3m42.954s 1.000 preempt 3m46.141s 3m45.835s 3m45.953s 1.010
Compute load comparisons are boring 'course.
-Mike
H. Peter Anvin
reviewerQuoted 20 lines · Mike Galbraith
Quoted 4 lines
Do you have any quantification of "munches throughput?" It seems odd that it would be worse than polling for preempt all over the kernel, but perhaps the additional locking is what costs.
I hadn't compared in ages, so made some fresh samples.
Q6600 3.11-rc4
vmark
voluntary 169808 155826 154741 1.000 preempt 149354 124016 128436 .836That should be ~worst case, it hates preemption.
tbench 8
voluntary 1027.96 1028.76 1044.60 1.000
preempt 929.06 935.01 928.64 .900hackbench -l 10000
Quoted 9 lines
voluntary 23.146 23.124 23.230 1.000 preempt 25.065 24.633 24.789 1.071kbuild vmlinux
voluntary 3m44.842s 3m42.975s 3m42.954s 1.000 preempt 3m46.141s 3m45.835s 3m45.953s 1.010Compute load comparisons are boring 'course.
I presume voluntary is indistinguishable from no preemption at all?
Either way, that is definitely a reproducible test case, so if someone is willing to take on optimizing preemption they can use vmark as the litmus test. It would be really awesome if we genuinely could get the cost of preemption down to where it just doesn't matter.
-hpa
Quoted 24 lines · H. Peter Anvin
Quoted 23 lines · Mike Galbraith
Quoted 4 lines
Do you have any quantification of "munches throughput?" It seems odd that it would be worse than polling for preempt all over the kernel, but perhaps the additional locking is what costs.
I hadn't compared in ages, so made some fresh samples.
Q6600 3.11-rc4
vmark
voluntary 169808 155826 154741 1.000 preempt 149354 124016 128436 .836That should be ~worst case, it hates preemption.
tbench 8
voluntary 1027.96 1028.76 1044.60 1.000
preempt 929.06 935.01 928.64 .900hackbench -l 10000
voluntary 23.146 23.124 23.230 1.000 preempt 25.065 24.633 24.789 1.071
Quoted 8 lines
Quoted 6 lines
kbuild vmlinux
voluntary 3m44.842s 3m42.975s 3m42.954s 1.000 preempt 3m46.141s 3m45.835s 3m45.953s 1.010Compute load comparisons are boring 'course.
I presume voluntary is indistinguishable from no preemption at all?
No, all preemption options produce performance deltas.
Quoted 4 lines
Either way, that is definitely a reproducible test case, so if someone is willing to take on optimizing preemption they can use vmark as the litmus test. It would be really awesome if we genuinely could get the cost of preemption down to where it just doesn't matter.
You have to eat more scheduler cycles, that's what PREEMPT does for a living. Release a lock, wham.
-Mike
H. Peter Anvin
reviewerThat sounds like an issue with specific preemption policies.
Quoted 49 lines · Mike Galbraith
Quoted 3 lines · H. Peter Anvin
Quoted 2 lines · Mike Galbraith
Quoted 2 lines
Do you have any quantification of "munches throughput?" It seems
odd
Quoted 1 line
Quoted 1 line
Quoted 1 line
that it would be worse than polling for preempt all over the
kernel, but
Quoted 20 lines
Quoted 20 lines
Quoted 1 line
perhaps the additional locking is what costs.
I hadn't compared in ages, so made some fresh samples.
Q6600 3.11-rc4
vmark
voluntary 169808 155826 154741 1.000 preempt 149354 124016 128436 .836That should be ~worst case, it hates preemption.
tbench 8
voluntary 1027.96 1028.76 1044.60 1.000
preempt 929.06 935.01 928.64 .900hackbench -l 10000
voluntary 23.146 23.124 23.230 1.000 preempt 25.065 24.633 24.789 1.071Quoted 8 lines
Quoted 6 lines
kbuild vmlinux
voluntary 3m44.842s 3m42.975s 3m42.954s 1.000 preempt 3m46.141s 3m45.835s 3m45.953s 1.010Compute load comparisons are boring 'course.
I presume voluntary is indistinguishable from no preemption at all?
No, all preemption options produce performance deltas.
Quoted 1 line
Either way, that is definitely a reproducible test case, so if
someone
Quoted 2 lines
is willing to take on optimizing preemption they can use vmark as the litmus test. It would be really awesome if we genuinely could get
the
Quoted 1 line
cost of preemption down to where it just doesn't matter.
You have to eat more scheduler cycles, that's what PREEMPT does for a living. Release a lock, wham.
-Mike
Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
Quoted 1 line · H. Peter Anvin
That sounds like an issue with specific preemption policies.
Actually, voluntary/nopreempt delta for _these_ loads was nil.
-Mike
H. Peter Anvin
reviewerQuoted 6 lines · Andi Kleen
The x86 user access functions (
*_user) were originally very well tuned, with partial inline code and other optimizations.Then over time various new checks -- particularly the sleep checks for a voluntary preempt kernel -- destroyed a lot of the tunings
Hi Andi,
Are you going to respin this patchset to address the feedback?
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 6 lines · Andi Kleen
The x86 user access functions (
*_user) were originally very well tuned, with partial inline code and other optimizations.Then over time various new checks -- particularly the sleep checks for a voluntary preempt kernel -- destroyed a lot of the tunings
Hi Andi,
Are you going to respin this patchset to address the feedback?
Yes. I'm dropping all the user checks.
But you could already merge the first patch, it's independent of all the others and not affected by the feedback.
-Andi
Quoted 22 lines · Andi Kleen
The x86 user access functions (
*_user) were originally very well tuned, with partial inline code and other optimizations.Then over time various new checks -- particularly the sleep checks for a voluntary preempt kernel -- destroyed a lot of the tunings
A typical user access operation is now doing multiple useless function calls. Also the without force inline gcc's inlining policy makes it even worse, with adding more unnecessary calls.
Here's a typical example from ftrace:
10) | might_fault() { 10) | _cond_resched() { 10) | should_resched() { 10) | need_resched() { 10) 0.063 us | test_ti_thread_flag(); 10) 0.643 us | } 10) 1.238 us | } 10) 1.845 us | } 10) 2.438 us | }
Quoted 2 lines
So we spent 2.5us doing nothing (ok it's a bit less without ftrace, but still pretty bad)
Hmm, which kernel version is this?
I thought I fixed this for good in
commit 114276ac0a3beb9c391a410349bd770653e185ce Author: Michael S. Tsirkin <[email protected]> Date: Sun May 26 17:32:13 2013 +0300 mm, sched: Drop voluntary schedule from might_fault()
might_fault shouldn't be calling cond_resched anymore.
Did this get reverted at some point?
I hope not, there's more code relying on this now (see e.g. 662bbcb2747c2422cf98d3d97619509379eee466)
Quoted 37 lines
Then in other cases we would have an out of line function, but would actually do the
might_sleep()checks in the inlined caller. This doesn't make any sense at all.There were also a few other problems, for example the x86-64 uaccess code regularly falls back to string functions, even though a simple mov would be enough. For example every futex access to the lock variable would actually use string instructions, even though it's just 4 bytes.
This patch kit is an attempt to get us back to sane code, mostly by doing proper inlining and doing sleep checks in the right place. Unfortunately I had to add one tree sweep to avoid an nasty include loop.
It costs a bit of text space, but I think it's worth it (if only to keep my blood pressure down while reading ftrace logs...)
I haven't done any particular benchmarks, but important low level functions just ought to be fast.
64bit:
13249492 1881328 1159168 16289988 f890c4 vmlinux-before-uaccess 13260877 1877232 1159168 16297277 f8ad3d vmlinux-uaccess+ 11k, +0.08%
32bit:
11223248 899512 1916928 14039688 d63a88 vmlinux-before-uaccess 11230358 895416 1916928 14042702 d6464e vmlinux-uaccess+ 7k, +0.06%
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines
I thought I fixed this for good in
commit 114276ac0a3beb9c391a410349bd770653e185ce Author: Michael S. Tsirkin <[email protected]> Date: Sun May 26 17:32:13 2013 +0300 mm, sched: Drop voluntary schedule from might_fault()
You're right this was an older kernel. So you already fixed part of it. Great.
I'll remove those parts.
-Andi