Re-tune x86 uaccess code for PREEMPT_VOLUNTARY v2

PATCH In review Andi Kleen 3 participants 11 messages Aug 14, 2013

Subsystems (62)

Networking Network drivers Memory management Arm (32-bit) +58 more subsystems

lore

Share & tools

lore

Andi Kleen

author

2013-08-14 00:09 UTC #0 lore raw

The x86 user access functions (*_user) were originally very well tuned, with partial inline code and other optimizations.

Then over time various new checks -- particularly the sleep checks for a voluntary preempt kernel -- destroyed a lot of the tunings

A typical user access operation is now doing multiple useless function calls. Also the without force inline gcc's inlining policy makes it even worse, with adding more unnecessary calls.

Here's a typical example from ftrace:

     10)               |    might_fault() {
     10)               |      _cond_resched() {
     10)               |        should_resched() {
     10)               |          need_resched() {
     10)   0.063 us    |            test_ti_thread_flag();
     10)   0.643 us    |          }
     10)   1.238 us    |        }
     10)   1.845 us    |      }
     10)   2.438 us    |    }

So we spent 2.5us doing nothing (ok it's a bit less without ftrace, but still pretty bad)

Then in other cases we would have an out of line function, but would actually do the might_sleep() checks in the inlined caller. This doesn't make any sense at all.

There were also a few other problems, for example the x86-64 uaccess code regularly falls back to string functions, even though a simple mov would be enough. For example every futex access to the lock variable would actually use string instructions, even though it's just 4 bytes.

This patch kit is an attempt to get us back to sane code, mostly by doing proper inlining and doing sleep checks in the right place. Unfortunately I had to add one tree sweep to avoid an nasty include loop.

v2: Now completely remove reschedule checks for uaccess functions.

Andi Kleen

author

2013-08-14 00:07 UTC #1 lore raw

PATCH 1/8 x86: Add 1/2/4/8 byte optimization to 64bit __copy_{from,to}_user_inatomic

From: Andi Kleen <[email protected]>

The 64bit __copy_{from,to}_user_inatomic always called copy_from_user_generic, but skipped the special optimizations for 1/2/4/8 byte accesses.

This especially hurts the futex call, which accesses the 4 byte futex user value with a complicated fast string operation in a function call, instead of a single movl.

Use __copy_{from,to}_user for _inatomic instead to get the same optimizations. The only problem was the might_fault() in those functions. So move that to new wrapper and call __copy_{f,t}_user_nocheck() from *_inatomic directly.

32bit already did this correctly by duplicating the code.

Signed-off-by Andi Kleen <[email protected]>

---
 arch/x86/include/asm/uaccess_64.h | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

arch/x86/include/asm/uaccess_64.h +18 -6

index 4f7923d..64476bb 100644
@@ -77,11 +77,10 @@ int copy_to_user(void __user *dst, const void *src, unsigned size)
 }
 
 static __always_inline __must_check
-int __copy_from_user(void *dst, const void __user *src, unsigned size)
+int __copy_from_user_nocheck(void *dst, const void __user *src, unsigned size)
 {
 	int ret = 0;
 
-	might_fault();
 	if (!__builtin_constant_p(size))
 		return copy_user_generic(dst, (__force void *)src, size);
 	switch (size) {

@@ -121,11 +120,17 @@ int __copy_from_user(void *dst, const void __user *src, unsigned size)
 }
 
 static __always_inline __must_check
-int __copy_to_user(void __user *dst, const void *src, unsigned size)
+int __copy_from_user(void *dst, const void __user *src, unsigned size)
+{
+	might_fault();
+	return __copy_from_user_nocheck(dst, src, size);
+}
+
+static __always_inline __must_check
+int __copy_to_user_nocheck(void __user *dst, const void *src, unsigned size)
 {
 	int ret = 0;
 
-	might_fault();
 	if (!__builtin_constant_p(size))
 		return copy_user_generic((__force void *)dst, src, size);
 	switch (size) {

@@ -165,6 +170,13 @@ int __copy_to_user(void __user *dst, const void *src, unsigned size)
 }
 
 static __always_inline __must_check
+int __copy_to_user(void __user *dst, const void *src, unsigned size)
+{
+	might_fault();
+	return __copy_to_user_nocheck(dst, src, size);
+}
+
+static __always_inline __must_check
 int __copy_in_user(void __user *dst, const void __user *src, unsigned size)
 {
 	int ret = 0;

@@ -220,13 +232,13 @@ int __copy_in_user(void __user *dst, const void __user *src, unsigned size)
 static __must_check __always_inline int
 __copy_from_user_inatomic(void *dst, const void __user *src, unsigned size)
 {
-	return copy_user_generic(dst, (__force const void *)src, size);
+	return __copy_from_user_nocheck(dst, (__force const void *)src, size);
 }
 
 static __must_check __always_inline int
 __copy_to_user_inatomic(void __user *dst, const void *src, unsigned size)
 {
-	return copy_user_generic((__force void *)dst, src, size);
+	return __copy_to_user_nocheck((__force void *)dst, src, size);
 }
 
 extern long __copy_user_nocache(void *dst, const void __user *src,

Signature · 1 line

1.8.3.1

Linus Torvalds

2013-08-14 00:17 UTC #2 lore raw

Re: [PATCH 1/8] x86: Add 1/2/4/8 byte optimization to 64bit __copy_{from,to}_user_inatomic

Quoted 2 lines · Andi Kleen

32bit already did this correctly by duplicating the code.

I wonder how much of this could be in asm/uaccess.h? We already do all the fixed-size get_user/put_user stuff in that generic x86 code, and I'm wondering why we don't do the "__builtin_constant_p()" cases of copy_to/from_user() there too?

It's independent of this patch-series, but it might be good to take a look at unifying these things..

Linus

Andi Kleen

author

2013-08-14 00:08 UTC #3 lore raw

PATCH 2/8 x86: Include linux/sched.h in asm/uaccess.h

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

uaccess.h uses might_sleep, but there is currently no explicit include for this. Since a upcoming patch moves might_sleep into sched.h include sched.h here.

Signed-off-by Andi Kleen <[email protected]>

---
 arch/x86/include/asm/uaccess.h | 1 +
 1 file changed, 1 insertion(+)

arch/x86/include/asm/uaccess.h +1 -0

index 5ee2687..8fa3bd6 100644
@@ -3,6 +3,7 @@
 /*
  * User space memory access functions
  */
+#include <linux/sched.h>
 #include <linux/errno.h>
 #include <linux/compiler.h>
 #include <linux/thread_info.h>

Signature · 1 line

1.8.3.1

Andi Kleen

author

2013-08-14 00:08 UTC #4 lore raw

PATCH 3/8 tree-sweep: Include linux/sched.h for might_sleep users

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

might_sleep is moving from linux/kernel.h to linux/sched.h, so any users need to include linux/sched.h

This was done with a mechanistic script and some uses may be redundant (already included in some other include file). However it's good practice to always include any needed symbols from the top level .c file.

Tested with x86-64 allyesconfig. I used to do a x86-32 allyesconfig on a old kernel, but since that is broken now I didn't retest.

Signed-off-by Andi Kleen <[email protected]>

---
 arch/arm/common/mcpm_entry.c                              | 1 +
 arch/arm/mach-omap2/omap_hwmod.c                          | 1 +
 arch/arm/mm/highmem.c                                     | 1 +
 arch/blackfin/kernel/bfin_gpio.c                          | 1 +
 arch/frv/mm/highmem.c                                     | 1 +
 arch/m32r/include/asm/uaccess.h                           | 1 +
 arch/microblaze/include/asm/highmem.h                     | 1 +
 arch/mn10300/include/asm/uaccess.h                        | 1 +
 arch/parisc/include/asm/cacheflush.h                      | 1 +
 arch/powerpc/include/asm/highmem.h                        | 1 +
 arch/powerpc/kernel/rtas.c                                | 1 +
 arch/powerpc/lib/checksum_wrappers_64.c                   | 1 +
 arch/powerpc/lib/usercopy_64.c                            | 1 +
 arch/tile/mm/highmem.c                                    | 1 +
 arch/x86/include/asm/checksum_32.h                        | 1 +
 arch/x86/lib/csum-wrappers_64.c                           | 1 +
 arch/x86/mm/highmem_32.c                                  | 1 +
 arch/x86/mm/mmio-mod.c                                    | 1 +
 block/blk-cgroup.c                                        | 1 +
 block/blk-core.c                                          | 1 +
 block/genhd.c                                             | 1 +
 drivers/base/dma-buf.c                                    | 1 +
 drivers/block/rsxx/dev.c                                  | 1 +
 drivers/dma/ipu/ipu_irq.c                                 | 1 +
 drivers/gpio/gpiolib.c                                    | 1 +
 drivers/ide/ide-io.c                                      | 1 +
 drivers/infiniband/hw/amso1100/c2_cq.c                    | 1 +
 drivers/infiniband/hw/cxgb3/iwch_cm.c                     | 1 +
 drivers/infiniband/hw/cxgb4/cm.c                          | 1 +
 drivers/md/dm.c                                           | 1 +
 drivers/md/raid5.c                                        | 1 +
 drivers/mmc/core/core.c                                   | 1 +
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c          | 1 +
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c            | 1 +
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.h         | 1 +
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c         | 1 +
 drivers/net/ethernet/intel/e1000e/netdev.c                | 1 +
 drivers/net/ethernet/intel/igbvf/netdev.c                 | 1 +
 drivers/net/ethernet/sfc/falcon.c                         | 1 +
 drivers/net/ieee802154/at86rf230.c                        | 1 +
 drivers/net/ieee802154/fakelb.c                           | 1 +
 drivers/net/wireless/ath/carl9170/usb.c                   | 1 +
 drivers/net/wireless/ath/wil6210/wmi.c                    | 1 +
 drivers/net/wireless/b43/dma.c                            | 1 +
 drivers/net/wireless/b43/main.c                           | 1 +
 drivers/net/wireless/b43/phy_a.c                          | 1 +
 drivers/net/wireless/b43/phy_g.c                          | 1 +
 drivers/net/wireless/b43legacy/dma.c                      | 1 +
 drivers/net/wireless/b43legacy/radio.c                    | 1 +
 drivers/net/wireless/cw1200/cw1200_spi.c                  | 1 +
 drivers/net/wireless/iwlwifi/dvm/sta.c                    | 1 +
 drivers/net/wireless/iwlwifi/iwl-op-mode.h                | 1 +
 drivers/net/wireless/iwlwifi/iwl-trans.h                  | 1 +
 drivers/net/wireless/libertas_tf/cmd.c                    | 1 +
 drivers/pci/iov.c                                         | 1 +
 drivers/pci/pci.c                                         | 1 +
 drivers/platform/olpc/olpc-ec.c                           | 1 +
 drivers/ssb/driver_pcicore.c                              | 1 +
 drivers/staging/lustre/lustre/llite/remote_perm.c         | 1 +
 drivers/staging/lustre/lustre/obdclass/cl_lock.c          | 1 +
 drivers/staging/lustre/lustre/obdclass/cl_object.c        | 1 +
 drivers/staging/lustre/lustre/obdclass/cl_page.c          | 1 +
 drivers/staging/lustre/lustre/osc/osc_lock.c              | 1 +
 drivers/staging/lustre/lustre/osc/osc_page.c              | 1 +
 drivers/staging/lustre/lustre/ptlrpc/client.c             | 1 +
 drivers/staging/lustre/lustre/ptlrpc/gss/gss_cli_upcall.c | 1 +
 drivers/staging/lustre/lustre/ptlrpc/gss/gss_pipefs.c     | 1 +
 drivers/staging/lustre/lustre/ptlrpc/sec.c                | 1 +
 drivers/staging/lustre/lustre/ptlrpc/sec_config.c         | 1 +
 drivers/staging/lustre/lustre/ptlrpc/sec_gc.c             | 1 +
 drivers/usb/core/hcd.c                                    | 1 +
 drivers/usb/core/urb.c                                    | 1 +
 drivers/video/atmel_lcdfb.c                               | 1 +
 fs/block_dev.c                                            | 1 +
 fs/buffer.c                                               | 1 +
 fs/dcache.c                                               | 1 +
 fs/ext3/inode.c                                           | 1 +
 fs/ext4/ext4_jbd2.c                                       | 1 +
 fs/ext4/inode.c                                           | 1 +
 fs/ext4/mballoc.c                                         | 1 +
 fs/file_table.c                                           | 1 +
 fs/inode.c                                                | 1 +
 fs/jbd/revoke.c                                           | 1 +
 fs/jbd2/revoke.c                                          | 1 +
 fs/locks.c                                                | 1 +
 fs/nfs/nfs4filelayoutdev.c                                | 1 +
 fs/nfs/nfs4proc.c                                         | 1 +
 fs/nfs/nfs4state.c                                        | 1 +
 fs/nilfs2/the_nilfs.c                                     | 1 +
 fs/xfs/xfs_mount.c                                        | 1 +
 include/asm-generic/gpio.h                                | 1 +
 include/linux/buffer_head.h                               | 1 +
 include/linux/clk.h                                       | 1 +
 include/linux/gpio.h                                      | 1 +
 include/linux/highmem.h                                   | 1 +
 include/linux/pagemap.h                                   | 1 +
 kernel/fork.c                                             | 1 +
 kernel/freezer.c                                          | 1 +
 kernel/irq/chip.c                                         | 1 +
 kernel/nsproxy.c                                          | 1 +
 kernel/printk/printk.c                                    | 1 +
 kernel/sched/core.c                                       | 1 +
 kernel/smp.c                                              | 1 +
 mm/hugetlb.c                                              | 1 +
 mm/memory.c                                               | 1 +
 mm/mmap.c                                                 | 1 +
 mm/rmap.c                                                 | 1 +
 net/caif/cfcnfg.c                                         | 1 +
 net/mac80211/driver-ops.h                                 | 1 +
 net/mac80211/key.c                                        | 1 +
 net/mac80211/main.c                                       | 1 +
 net/mac80211/sta_info.c                                   | 1 +
 net/phonet/pep.c                                          | 1 +
 net/sunrpc/clnt.c                                         | 1 +
 net/wimax/op-msg.c                                        | 1 +
 net/wimax/op-reset.c                                      | 1 +
 net/wimax/op-rfkill.c                                     | 1 +
 net/wireless/wext-proc.c                                  | 1 +
 sound/core/info.c                                         | 1 +
 virt/kvm/async_pf.c                                       | 1 +
 120 files changed, 120 insertions(+)

arch/arm/common/mcpm_entry.c +1 -0

index 370236d..c083f90 100644
@@ -10,6 +10,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/init.h>
 #include <linux/irqflags.h>

arch/arm/mach-omap2/omap_hwmod.c +1 -0

index 7f4db12..04a2674 100644
@@ -127,6 +127,7 @@
  */
 #undef DEBUG
 
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <linux/io.h>

arch/arm/mm/highmem.c +1 -0

index 21b9e1b..a8be1f1 100644
@@ -10,6 +10,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/highmem.h>
 #include <linux/interrupt.h>

arch/blackfin/kernel/bfin_gpio.c +1 -0

index ed978f1..0efe3d5 100644
@@ -6,6 +6,7 @@
  * Licensed under the GPL-2 or later
  */
 
+#include <linux/sched.h>
 #include <linux/delay.h>
 #include <linux/module.h>
 #include <linux/err.h>

arch/frv/mm/highmem.c +1 -0

index bed9a9b..766ff5b 100644
@@ -8,6 +8,7 @@
  * as published by the Free Software Foundation; either version
  * 2 of the License, or (at your option) any later version.
  */
+#include <linux/sched.h>
 #include <linux/highmem.h>
 #include <linux/module.h>

arch/m32r/include/asm/uaccess.h +1 -0

index 84fe7ba..a097157 100644
@@ -11,6 +11,7 @@
 /*
  * User space memory access functions
  */
+#include <linux/sched.h>
 #include <linux/errno.h>
 #include <linux/thread_info.h>
 #include <asm/page.h>

arch/microblaze/include/asm/highmem.h +1 -0

index d046389..40c5b59 100644
@@ -20,6 +20,7 @@
 #ifdef __KERNEL__
 
 #include <linux/init.h>
+#include <linux/sched.h>
 #include <linux/interrupt.h>
 #include <linux/uaccess.h>
 #include <asm/fixmap.h>

arch/mn10300/include/asm/uaccess.h +1 -0

index 5372787..274c9c2 100644
@@ -14,6 +14,7 @@
 /*
  * User space memory access functions
  */
+#include <linux/sched.h>
 #include <linux/thread_info.h>
 #include <linux/kernel.h>
 #include <asm/page.h>

arch/parisc/include/asm/cacheflush.h +1 -0

index f0e2784..ab4ed76 100644
@@ -2,6 +2,7 @@
 #define _PARISC_CACHEFLUSH_H
 
 #include <linux/mm.h>
+#include <linux/sched.h>
 #include <linux/uaccess.h>
 #include <asm/tlbflush.h>

arch/powerpc/include/asm/highmem.h +1 -0

index caaf6e0..721bc1b 100644
@@ -22,6 +22,7 @@
 
 #ifdef __KERNEL__
 
+#include <linux/sched.h>
 #include <linux/interrupt.h>
 #include <asm/kmap_types.h>
 #include <asm/tlbflush.h>

arch/powerpc/kernel/rtas.c +1 -0

index 80b5ef4..e479bcf 100644
@@ -11,6 +11,7 @@
  *      2 of the License, or (at your option) any later version.
  */
 
+#include <linux/sched.h>
 #include <stdarg.h>
 #include <linux/kernel.h>
 #include <linux/types.h>

arch/powerpc/lib/checksum_wrappers_64.c +1 -0

index 08e3a33..8d9598c 100644
@@ -17,6 +17,7 @@
  *
  * Author: Anton Blanchard <[email protected]>
  */
+#include <linux/sched.h>
 #include <linux/export.h>
 #include <linux/compiler.h>
 #include <linux/types.h>

arch/powerpc/lib/usercopy_64.c +1 -0

index 5eea6f3..8c8cfa6 100644
@@ -6,6 +6,7 @@
  * as published by the Free Software Foundation; either version
  * 2 of the License, or (at your option) any later version.
  */
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <asm/uaccess.h>

arch/tile/mm/highmem.c +1 -0

index 347d123..f82b1e0 100644
@@ -12,6 +12,7 @@
  *   more details.
  */
 
+#include <linux/sched.h>
 #include <linux/highmem.h>
 #include <linux/module.h>
 #include <linux/pagemap.h>

arch/x86/include/asm/checksum_32.h +1 -0

index 46fc474..b9aa5d0 100644
@@ -2,6 +2,7 @@
 #define _ASM_X86_CHECKSUM_32_H
 
 #include <linux/in6.h>
+#include <linux/sched.h>
 
 #include <asm/uaccess.h>

arch/x86/lib/csum-wrappers_64.c +1 -0

index 25b7ae8..aaba241 100644
@@ -4,6 +4,7 @@
  *
  * Wrappers of assembly checksum functions for x86-64.
  */
+#include <linux/sched.h>
 #include <asm/checksum.h>
 #include <linux/module.h>

arch/x86/mm/highmem_32.c +1 -0

index 4500142..1212a56 100644
@@ -1,3 +1,4 @@
+#include <linux/sched.h>
 #include <linux/highmem.h>
 #include <linux/module.h>
 #include <linux/swap.h> /* for totalram_pages */

arch/x86/mm/mmio-mod.c +1 -0

index 0057a7a..07b0235 100644
@@ -24,6 +24,7 @@
 
 #define DEBUG 1
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/debugfs.h>
 #include <linux/slab.h>

block/blk-cgroup.c +1 -0

index 290792a..20664e2 100644
@@ -11,6 +11,7 @@
  * 	              Nauman Rafique <[email protected]>
  */
 #include <linux/ioprio.h>
+#include <linux/sched.h>
 #include <linux/kdev_t.h>
 #include <linux/module.h>
 #include <linux/err.h>

block/blk-core.c +1 -0

index 93a18d1..cf9bb93 100644
@@ -11,6 +11,7 @@
 /*
  * This handles all read/write requests to block devices
  */
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/backing-dev.h>

block/genhd.c +1 -0

index dadf42b..a5012aa9 100644
@@ -2,6 +2,7 @@
  *  gendisk handling
  */
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/fs.h>
 #include <linux/genhd.h>

drivers/base/dma-buf.c +1 -0

index 6687ba7..b116b28 100644
@@ -22,6 +22,7 @@
  * this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include <linux/sched.h>
 #include <linux/fs.h>
 #include <linux/slab.h>
 #include <linux/dma-buf.h>

drivers/block/rsxx/dev.c +1 -0

index d7af441..a033bce 100644
@@ -23,6 +23,7 @@
 */
 
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>

drivers/dma/ipu/ipu_irq.c +1 -0

index 2e284a4..7f4b2e5 100644
@@ -7,6 +7,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/sched.h>
 #include <linux/init.h>
 #include <linux/err.h>
 #include <linux/spinlock.h>

drivers/gpio/gpiolib.c +1 -0

index ff0fd65..906a80d 100644
@@ -1,3 +1,4 @@
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/interrupt.h>

drivers/ide/ide-io.c +1 -0

index 177db6d..9aa758e 100644
@@ -24,6 +24,7 @@
  */
  
  
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/types.h>
 #include <linux/string.h>

drivers/infiniband/hw/amso1100/c2_cq.c +1 -0

index 49e0e85..07f9d3a 100644
@@ -35,6 +35,7 @@
  * SOFTWARE.
  *
  */
+#include <linux/sched.h>
 #include <linux/gfp.h>
 
 #include "c2.h"

drivers/infiniband/hw/cxgb3/iwch_cm.c +1 -0

index 3e094cd..a0fba9c 100644
@@ -29,6 +29,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  */
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/list.h>
 #include <linux/slab.h>

drivers/infiniband/hw/cxgb4/cm.c +1 -0

index 65c30ea..f0e6a5b 100644
@@ -29,6 +29,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  */
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/list.h>
 #include <linux/workqueue.h>

drivers/md/dm.c +1 -0

index 9e39d2b..43ee49a 100644
@@ -5,6 +5,7 @@
  * This file is released under the GPL.
  */
 
+#include <linux/sched.h>
 #include "dm.h"
 #include "dm-uevent.h"

drivers/md/raid5.c +1 -0

index 78ea443..fcf17e7 100644
@@ -43,6 +43,7 @@
  * miss any bits.
  */
 
+#include <linux/sched.h>
 #include <linux/blkdev.h>
 #include <linux/kthread.h>
 #include <linux/raid/pq.h>

drivers/mmc/core/core.c +1 -0

index 49a5bca..f1832b0 100644
@@ -10,6 +10,7 @@
  * it under the terms of the GNU General Public License version 2 as
  * published by the Free Software Foundation.
  */
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/interrupt.h>

drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +1 -0

index e06186c..95eed3f 100644
@@ -17,6 +17,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
 #include <linux/kernel.h>

drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c +1 -0

index 8f03c98..d908719 100644
@@ -19,6 +19,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/crc32.h>
 #include <linux/netdevice.h>

drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.h +1 -0

index d143a7c..01ebe72 100644
@@ -20,6 +20,7 @@
 #define BNX2X_SRIOV_H
 
 #include "bnx2x_vfpf.h"
+#include <linux/sched.h>
 #include "bnx2x.h"
 
 enum sample_bulletin_result {

drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c +1 -0

index 98366ab..962e22a 100644
@@ -17,6 +17,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/sched.h>
 #include "bnx2x_stats.h"
 #include "bnx2x_cmn.h"
 #include "bnx2x_sriov.h"

drivers/net/ethernet/intel/e1000e/netdev.c +1 -0

index 77f81cb..36aca7e 100644
@@ -28,6 +28,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/types.h>
 #include <linux/init.h>

drivers/net/ethernet/intel/igbvf/netdev.c +1 -0

index 93eb7ee..95d5430 100644
@@ -27,6 +27,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/types.h>
 #include <linux/init.h>

drivers/net/ethernet/sfc/falcon.c +1 -0

index 71998e7..a7a390f 100644
@@ -8,6 +8,7 @@
  * by the Free Software Foundation, incorporated herein by reference.
  */
 
+#include <linux/sched.h>
 #include <linux/bitops.h>
 #include <linux/delay.h>
 #include <linux/pci.h>

drivers/net/ieee802154/at86rf230.c +1 -0

index 6f10b49..59350e3 100644
@@ -21,6 +21,7 @@
  * Alexander Smirnov <[email protected]>
  */
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/interrupt.h>
 #include <linux/gpio.h>

drivers/net/ieee802154/fakelb.c +1 -0

index b8d2217..124969c 100644
@@ -23,6 +23,7 @@
  */
 
 #include <linux/module.h>
+#include <linux/sched.h>
 #include <linux/timer.h>
 #include <linux/platform_device.h>
 #include <linux/netdevice.h>

drivers/net/wireless/ath/carl9170/usb.c +1 -0

index 307bc0d..a6bc868 100644
@@ -37,6 +37,7 @@
  *    OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  */
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/usb.h>

drivers/net/wireless/ath/wil6210/wmi.c +1 -0

index dc8059a..bc61ab3 100644
@@ -15,6 +15,7 @@
  */
 
 #include <linux/etherdevice.h>
+#include <linux/sched.h>
 #include <linux/if_arp.h>
 
 #include "wil6210.h"

drivers/net/wireless/b43/dma.c +1 -0

index f7c70b3..2bdbd63 100644
@@ -27,6 +27,7 @@
 
 */
 
+#include <linux/sched.h>
 #include "b43.h"
 #include "dma.h"
 #include "main.h"

drivers/net/wireless/b43/main.c +1 -0

index 0e933bb..6e49c09 100644
@@ -32,6 +32,7 @@
 
 */
 
+#include <linux/sched.h>
 #include <linux/delay.h>
 #include <linux/init.h>
 #include <linux/module.h>

drivers/net/wireless/b43/phy_a.c +1 -0

index a6c3810..0a7d6b6 100644
@@ -26,6 +26,7 @@
 
 */
 
+#include <linux/sched.h>
 #include <linux/slab.h>
 
 #include "b43.h"

drivers/net/wireless/b43/phy_g.c +1 -0

index 12f467b..01141cd 100644
@@ -26,6 +26,7 @@
 
 */
 
+#include <linux/sched.h>
 #include "b43.h"
 #include "phy_g.h"
 #include "phy_common.h"

drivers/net/wireless/b43legacy/dma.c +1 -0

index faeafe2..f709415 100644
@@ -27,6 +27,7 @@
 
 */
 
+#include <linux/sched.h>
 #include "b43legacy.h"
 #include "dma.h"
 #include "main.h"

drivers/net/wireless/b43legacy/radio.c +1 -0

index 8961776..ff59e7e 100644
@@ -29,6 +29,7 @@
 
 */
 
+#include <linux/sched.h>
 #include <linux/delay.h>
 
 #include "b43legacy.h"

drivers/net/wireless/cw1200/cw1200_spi.c +1 -0

index d063760..c2795d4 100644
@@ -14,6 +14,7 @@
  */
 
 #include <linux/module.h>
+#include <linux/sched.h>
 #include <linux/gpio.h>
 #include <linux/delay.h>
 #include <linux/spinlock.h>

drivers/net/wireless/iwlwifi/dvm/sta.c +1 -0

index c3c13ce..71df3ce 100644
@@ -27,6 +27,7 @@
  *
  *****************************************************************************/
 #include <linux/etherdevice.h>
+#include <linux/sched.h>
 #include <net/mac80211.h>
 #include "iwl-trans.h"
 #include "dev.h"

drivers/net/wireless/iwlwifi/iwl-op-mode.h +1 -0

index 98c7aa7..92695a6 100644
@@ -64,6 +64,7 @@
 #define __iwl_op_mode_h__
 
 #include <linux/debugfs.h>
+#include <linux/sched.h>
 
 struct iwl_op_mode;
 struct iwl_trans;

drivers/net/wireless/iwlwifi/iwl-trans.h +1 -0

index 8d91422c..231949f 100644
@@ -63,6 +63,7 @@
 #ifndef __iwl_trans_h__
 #define __iwl_trans_h__
 
+#include <linux/sched.h>
 #include <linux/ieee80211.h>
 #include <linux/mm.h> /* for page_address */
 #include <linux/lockdep.h>

drivers/net/wireless/libertas_tf/cmd.c +1 -0

index 909ac36..fa69e9f 100644
@@ -9,6 +9,7 @@
  */
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/sched.h>
 #include <linux/hardirq.h>
 #include <linux/slab.h>
 #include <linux/export.h>

drivers/pci/iov.c +1 -0

index de8ffac..a581e34 100644
@@ -8,6 +8,7 @@
  *   Address Translation Service 1.0
  */
 
+#include <linux/sched.h>
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/mutex.h>

drivers/pci/pci.c +1 -0

index e37fea6..6b44f63 100644
@@ -7,6 +7,7 @@
  *	Copyright 1997 -- 2000 Martin Mares <[email protected]>
  */
 
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/delay.h>
 #include <linux/init.h>

drivers/platform/olpc/olpc-ec.c +1 -0

index 0f9f859..dd3984a 100644
@@ -6,6 +6,7 @@
  * Licensed under the GPL v2 or later.
  */
 #include <linux/completion.h>
+#include <linux/sched.h>
 #include <linux/debugfs.h>
 #include <linux/spinlock.h>
 #include <linux/mutex.h>

drivers/ssb/driver_pcicore.c +1 -0

index d75b72b..c343ee7 100644
@@ -8,6 +8,7 @@
  * Licensed under the GNU/GPL. See COPYING for details.
  */
 
+#include <linux/sched.h>
 #include <linux/ssb/ssb.h>
 #include <linux/pci.h>
 #include <linux/export.h>

drivers/staging/lustre/lustre/llite/remote_perm.c +1 -0

index 68b2dc4..ceac936 100644
@@ -44,6 +44,7 @@
 #define DEBUG_SUBSYSTEM S_LLITE
 
 #include <linux/module.h>
+#include <linux/sched.h>
 #include <linux/types.h>
 #include <linux/version.h>

drivers/staging/lustre/lustre/obdclass/cl_lock.c +1 -0

index d34e044..3fb55ff 100644
@@ -41,6 +41,7 @@
 #define DEBUG_SUBSYSTEM S_CLASS
 
 #include <obd_class.h>
+#include <linux/sched.h>
 #include <obd_support.h>
 #include <lustre_fid.h>
 #include <linux/list.h>

drivers/staging/lustre/lustre/obdclass/cl_object.c +1 -0

index cdb5fba..72a1ac4 100644
@@ -52,6 +52,7 @@
 #define DEBUG_SUBSYSTEM S_CLASS
 
 #include <linux/libcfs/libcfs.h>
+#include <linux/sched.h>
 /* class_put_type() */
 #include <obd_class.h>
 #include <obd_support.h>

drivers/staging/lustre/lustre/obdclass/cl_page.c +1 -0

index bb93359..f5189f8 100644
@@ -41,6 +41,7 @@
 #define DEBUG_SUBSYSTEM S_CLASS
 
 #include <linux/libcfs/libcfs.h>
+#include <linux/sched.h>
 #include <obd_class.h>
 #include <obd_support.h>
 #include <linux/list.h>

drivers/staging/lustre/lustre/osc/osc_lock.c +1 -0

index 640bc3d..d38347b 100644
@@ -43,6 +43,7 @@
 # include <linux/libcfs/libcfs.h>
 /* fid_build_reg_res_name() */
 #include <lustre_fid.h>
+#include <linux/sched.h>
 
 #include "osc_cl_internal.h"

drivers/staging/lustre/lustre/osc/osc_page.c +1 -0

index baba959..ad2c6c7 100644
@@ -41,6 +41,7 @@
 #define DEBUG_SUBSYSTEM S_OSC
 
 #include "osc_cl_internal.h"
+#include <linux/sched.h>
 
 static void osc_lru_del(struct client_obd *cli, struct osc_page *opg, bool del);
 static void osc_lru_add(struct client_obd *cli, struct osc_page *opg);

drivers/staging/lustre/lustre/ptlrpc/client.c +1 -0

index 22f7e65..6791b65 100644
@@ -39,6 +39,7 @@
 #define DEBUG_SUBSYSTEM S_RPC
 
 #include <obd_support.h>
+#include <linux/sched.h>
 #include <obd_class.h>
 #include <lustre_lib.h>
 #include <lustre_ha.h>

drivers/staging/lustre/lustre/ptlrpc/gss/gss_cli_upcall.c +1 -0

index 142c789..b2b893c 100644
@@ -40,6 +40,7 @@
 
 #define DEBUG_SUBSYSTEM S_SEC
 #include <linux/init.h>
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/dcache.h>

drivers/staging/lustre/lustre/ptlrpc/gss/gss_pipefs.c +1 -0

index 3df7257..9dddaa8 100644
@@ -48,6 +48,7 @@
 
 #define DEBUG_SUBSYSTEM S_SEC
 #include <linux/init.h>
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/dcache.h>

drivers/staging/lustre/lustre/ptlrpc/sec.c +1 -0

index 36e8bed..28dc4e6 100644
@@ -41,6 +41,7 @@
 #define DEBUG_SUBSYSTEM S_SEC
 
 #include <linux/libcfs/libcfs.h>
+#include <linux/sched.h>
 #include <linux/crypto.h>
 #include <linux/key.h>

drivers/staging/lustre/lustre/ptlrpc/sec_config.c +1 -0

index a45a392..c4b206c 100644
@@ -37,6 +37,7 @@
 #define DEBUG_SUBSYSTEM S_SEC
 
 #include <linux/libcfs/libcfs.h>
+#include <linux/sched.h>
 #include <linux/crypto.h>
 #include <linux/key.h>

drivers/staging/lustre/lustre/ptlrpc/sec_gc.c +1 -0

index 4c96a14a..b512cbc 100644
@@ -41,6 +41,7 @@
 #define DEBUG_SUBSYSTEM S_SEC
 
 #include <linux/libcfs/libcfs.h>
+#include <linux/sched.h>
 
 #include <obd_support.h>
 #include <obd_class.h>

drivers/usb/core/hcd.c +1 -0

index 014dc99..ae84867 100644
@@ -23,6 +23,7 @@
  */
 
 #include <linux/bcd.h>
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/version.h>
 #include <linux/kernel.h>

drivers/usb/core/urb.c +1 -0

index 16927fa..f033bcd0 100644
@@ -1,3 +1,4 @@
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/string.h>
 #include <linux/bitops.h>

drivers/video/atmel_lcdfb.c +1 -0

index effdb37..915836b 100644
@@ -8,6 +8,7 @@
  * more details.
  */
 
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/platform_device.h>
 #include <linux/dma-mapping.h>

fs/block_dev.c +1 -0

index c7bda5c..00f49af 100644
@@ -5,6 +5,7 @@
  *  Copyright (C) 2001  Andrea Arcangeli <[email protected]> SuSE
  */
 
+#include <linux/sched.h>
 #include <linux/init.h>
 #include <linux/mm.h>
 #include <linux/fcntl.h>

fs/buffer.c +1 -0

index 4d74335..050c03b 100644
@@ -18,6 +18,7 @@
  * async buffer flushing, 1999 Andrea Arcangeli <[email protected]>
  */
 
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/syscalls.h>
 #include <linux/fs.h>

fs/dcache.c +1 -0

index 87bdb53..ab41273 100644
@@ -14,6 +14,7 @@
  * the dcache entry is deleted or garbage collected.
  */
 
+#include <linux/sched.h>
 #include <linux/syscalls.h>
 #include <linux/string.h>
 #include <linux/mm.h>

fs/ext3/inode.c +1 -0

index 2bd8548..8caf634 100644
@@ -22,6 +22,7 @@
  *  Assorted race fixes, rewrite of ext3_get_block() by Al Viro, 2000
  */
 
+#include <linux/sched.h>
 #include <linux/highuid.h>
 #include <linux/quotaops.h>
 #include <linux/writeback.h>

fs/ext4/ext4_jbd2.c +1 -0

index 72a3600..c94cb65 100644
@@ -2,6 +2,7 @@
  * Interface between ext4 and JBD
  */
 
+#include <linux/sched.h>
 #include "ext4_jbd2.h"
 
 #include <trace/events/ext4.h>

fs/ext4/inode.c +1 -0

index dd32a2e..d12f990 100644
@@ -18,6 +18,7 @@
  *  Assorted race fixes, rewrite of ext4_get_block() by Al Viro, 2000
  */
 
+#include <linux/sched.h>
 #include <linux/fs.h>
 #include <linux/time.h>
 #include <linux/jbd2.h>

fs/ext4/mballoc.c +1 -0

index 4bbbf13b..108e7f6 100644
@@ -22,6 +22,7 @@
  */
 
 #include "ext4_jbd2.h"
+#include <linux/sched.h>
 #include "mballoc.h"
 #include <linux/log2.h>
 #include <linux/module.h>

fs/file_table.c +1 -0

index b44e4c5..b35edc4 100644
@@ -5,6 +5,7 @@
  *  Copyright (C) 1997 David S. Miller ([email protected])
  */
 
+#include <linux/sched.h>
 #include <linux/string.h>
 #include <linux/slab.h>
 #include <linux/file.h>

fs/inode.c +1 -0

index d6dfb09..ef3a12d 100644
@@ -2,6 +2,7 @@
  * (C) 1997 Linus Torvalds
  * (C) 1999 Andrea Arcangeli <[email protected]> (dynamic inode allocation)
  */
+#include <linux/sched.h>
 #include <linux/export.h>
 #include <linux/fs.h>
 #include <linux/mm.h>

fs/jbd/revoke.c +1 -0

index 25c713e..4642e55 100644
@@ -81,6 +81,7 @@
  */
 
 #ifndef __KERNEL__
+#include <linux/sched.h>
 #include "jfs_user.h"
 #else
 #include <linux/time.h>

fs/jbd2/revoke.c +1 -0

index 198c9c1..b406894 100644
@@ -81,6 +81,7 @@
  */
 
 #ifndef __KERNEL__
+#include <linux/sched.h>
 #include "jfs_user.h"
 #else
 #include <linux/time.h>

fs/locks.c +1 -0

index b27a300..b8a6709 100644
@@ -114,6 +114,7 @@
  *  Stephen Rothwell <[email protected]>, June, 2000.
  */
 
+#include <linux/sched.h>
 #include <linux/capability.h>
 #include <linux/file.h>
 #include <linux/fdtable.h>

fs/nfs/nfs4filelayoutdev.c +1 -0

index 95604f6..c26be3b 100644
@@ -29,6 +29,7 @@
  */
 
 #include <linux/nfs_fs.h>
+#include <linux/sched.h>
 #include <linux/vmalloc.h>
 #include <linux/module.h>
 #include <linux/sunrpc/addr.h>

fs/nfs/nfs4proc.c +1 -0

index cf11799..90319a3 100644
@@ -35,6 +35,7 @@
  *  SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
+#include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/delay.h>
 #include <linux/errno.h>

fs/nfs/nfs4state.c +1 -0

index e22862f..c9acdbc 100644
@@ -39,6 +39,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/slab.h>
 #include <linux/fs.h>
 #include <linux/nfs_fs.h>

fs/nilfs2/the_nilfs.c +1 -0

index 94c451c..462be53 100644
@@ -21,6 +21,7 @@
  *
  */
 
+#include <linux/sched.h>
 #include <linux/buffer_head.h>
 #include <linux/slab.h>
 #include <linux/blkdev.h>

fs/xfs/xfs_mount.c +1 -0

index 2b0ba35..eb9ba15 100644
@@ -15,6 +15,7 @@
  * along with this program; if not, write the Free Software Foundation,
  * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
  */
+#include <linux/sched.h>
 #include "xfs.h"
 #include "xfs_fs.h"
 #include "xfs_types.h"

include/asm-generic/gpio.h +1 -0

index bde6469..17114e0 100644
@@ -2,6 +2,7 @@
 #define _ASM_GENERIC_GPIO_H
 
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/types.h>
 #include <linux/errno.h>
 #include <linux/of.h>

include/linux/buffer_head.h +1 -0

index 91fa9a9..432e212 100644
@@ -8,6 +8,7 @@
 #define _LINUX_BUFFER_HEAD_H
 
 #include <linux/types.h>
+#include <linux/sched.h>
 #include <linux/fs.h>
 #include <linux/linkage.h>
 #include <linux/pagemap.h>

include/linux/clk.h +1 -0

index 9a6d045..8b440a2 100644
@@ -14,6 +14,7 @@
 
 #include <linux/err.h>
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/notifier.h>
 
 struct device;

include/linux/gpio.h +1 -0

index 552e3f4..be83d05 100644
@@ -2,6 +2,7 @@
 #define __LINUX_GPIO_H
 
 #include <linux/errno.h>
+#include <linux/sched.h>
 
 /* see Documentation/gpio.txt */

include/linux/highmem.h +1 -0

index 7fb31da..435de87 100644
@@ -2,6 +2,7 @@
 #define _LINUX_HIGHMEM_H
 
 #include <linux/fs.h>
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/bug.h>
 #include <linux/mm.h>

include/linux/pagemap.h +1 -0

index e3dea75..a409785 100644
@@ -5,6 +5,7 @@
  * Copyright 1995 Linus Torvalds
  */
 #include <linux/mm.h>
+#include <linux/sched.h>
 #include <linux/fs.h>
 #include <linux/list.h>
 #include <linux/highmem.h>

kernel/fork.c +1 -0

index 403d2bb..f12e417 100644
@@ -11,6 +11,7 @@
  * management can be a bitch. See 'mm/memory.c': 'copy_page_range()'
  */
 
+#include <linux/sched.h>
 #include <linux/slab.h>
 #include <linux/init.h>
 #include <linux/unistd.h>

kernel/freezer.c +1 -0

index b462fa1..1d975d1 100644
@@ -4,6 +4,7 @@
  * Originally from kernel/power/process.c
  */
 
+#include <linux/sched.h>
 #include <linux/interrupt.h>
 #include <linux/suspend.h>
 #include <linux/export.h>

kernel/irq/chip.c +1 -0

index a3bb14f..9948f28 100644
@@ -10,6 +10,7 @@
  * Detailed information is available in Documentation/DocBook/genericirq
  */
 
+#include <linux/sched.h>
 #include <linux/irq.h>
 #include <linux/msi.h>
 #include <linux/module.h>

kernel/nsproxy.c +1 -0

index 364ceab..d018644 100644
@@ -13,6 +13,7 @@
  *             Pavel Emelianov <[email protected]>
  */
 
+#include <linux/sched.h>
 #include <linux/slab.h>
 #include <linux/export.h>
 #include <linux/nsproxy.h>

kernel/printk/printk.c +1 -0

index 5b5a708..c97c144 100644
@@ -17,6 +17,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/tty.h>
 #include <linux/tty_driver.h>

kernel/sched/core.c +1 -0

index b7c32cb..74d7c04 100644
@@ -26,6 +26,7 @@
  *              Thomas Gleixner, Mike Kravetz
  */
 
+#include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/nmi.h>

kernel/smp.c +1 -0

index fe9f773..5b58226 100644
@@ -9,6 +9,7 @@
 #include <linux/export.h>
 #include <linux/percpu.h>
 #include <linux/init.h>
+#include <linux/sched.h>
 #include <linux/gfp.h>
 #include <linux/smp.h>
 #include <linux/cpu.h>

mm/hugetlb.c +1 -0

index 83aff0a..e342674 100644
@@ -2,6 +2,7 @@
  * Generic hugetlb support.
  * (C) Nadia Yvette Chambers, April 2004
  */
+#include <linux/sched.h>
 #include <linux/list.h>
 #include <linux/init.h>
 #include <linux/module.h>

mm/memory.c +1 -0

index 1ce2e2a..61add89 100644
@@ -38,6 +38,7 @@
  * Aug/Sep 2004 Changed to four level page tables (Andi Kleen)
  */
 
+#include <linux/sched.h>
 #include <linux/kernel_stat.h>
 #include <linux/mm.h>
 #include <linux/hugetlb.h>

mm/mmap.c +1 -0

index 1edbaa3..1fda64c 100644
@@ -7,6 +7,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/sched.h>
 #include <linux/slab.h>
 #include <linux/backing-dev.h>
 #include <linux/mm.h>

mm/rmap.c +1 -0

index cd356df..ca1c188 100644
@@ -42,6 +42,7 @@
  *     pte map lock
  */
 
+#include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/pagemap.h>
 #include <linux/swap.h>

net/caif/cfcnfg.c +1 -0

index fa39fc2..2809fd5 100644
@@ -6,6 +6,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ":%s(): " fmt, __func__
 
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/stddef.h>
 #include <linux/slab.h>

net/mac80211/driver-ops.h +1 -0

index b931c96..f6de52c 100644
@@ -1,6 +1,7 @@
 #ifndef __MAC80211_DRIVER_OPS
 #define __MAC80211_DRIVER_OPS
 
+#include <linux/sched.h>
 #include <net/mac80211.h>
 #include "ieee80211_i.h"
 #include "trace.h"

net/mac80211/key.c +1 -0

index e39cc91..9ae77d9 100644
@@ -9,6 +9,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/sched.h>
 #include <linux/if_ether.h>
 #include <linux/etherdevice.h>
 #include <linux/list.h>

net/mac80211/main.c +1 -0

index 091088a..53db5ee 100644
@@ -8,6 +8,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/sched.h>
 #include <net/mac80211.h>
 #include <linux/module.h>
 #include <linux/init.h>

net/mac80211/sta_info.c +1 -0

index aeb967a..5a5ae34 100644
@@ -7,6 +7,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/etherdevice.h>

net/phonet/pep.c +1 -0

index e774117..76bb2dc 100644
@@ -22,6 +22,7 @@
  * 02110-1301 USA
  */
 
+#include <linux/sched.h>
 #include <linux/kernel.h>
 #include <linux/slab.h>
 #include <linux/socket.h>

net/sunrpc/clnt.c +1 -0

index 74f6a70..6efea48 100644
@@ -19,6 +19,7 @@
 
 
 #include <linux/module.h>
+#include <linux/sched.h>
 #include <linux/types.h>
 #include <linux/kallsyms.h>
 #include <linux/mm.h>

net/wimax/op-msg.c +1 -0

index 0694d62..2d48d26 100644
@@ -71,6 +71,7 @@
  *   wimax_msg_alloc()
  *   wimax_msg_send()
  */
+#include <linux/sched.h>
 #include <linux/device.h>
 #include <linux/slab.h>
 #include <net/genetlink.h>

net/wimax/op-reset.c +1 -0

index 7ceffe3..30af0ee 100644
@@ -28,6 +28,7 @@
  * disconnect and reconnect the device).
  */
 
+#include <linux/sched.h>
 #include <net/wimax.h>
 #include <net/genetlink.h>
 #include <linux/wimax.h>

net/wimax/op-rfkill.c +1 -0

index 7ab60ba..6efe0bd 100644
@@ -60,6 +60,7 @@
  * wimax_rfkill_rm()            [called by wimax_dev_add/rm()]
  */
 
+#include <linux/sched.h>
 #include <net/wimax.h>
 #include <net/genetlink.h>
 #include <linux/wimax.h>

net/wireless/wext-proc.c +1 -0

index e98a01c..e771c39 100644
@@ -16,6 +16,7 @@
  * The content of the file is basically the content of "struct iw_statistics".
  */
 
+#include <linux/sched.h>
 #include <linux/module.h>
 #include <linux/proc_fs.h>
 #include <linux/seq_file.h>

sound/core/info.c +1 -0

index e79baa1..719a8ef 100644
@@ -19,6 +19,7 @@
  *
  */
 
+#include <linux/sched.h>
 #include <linux/init.h>
 #include <linux/time.h>
 #include <linux/mm.h>

virt/kvm/async_pf.c +1 -0

index ea475cd..b44cea0 100644
@@ -20,6 +20,7 @@
  * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.
  */
 
+#include <linux/sched.h>
 #include <linux/kvm_host.h>
 #include <linux/slab.h>
 #include <linux/module.h>

Signature · 1 line

1.8.3.1

Andi Kleen

author

2013-08-14 00:09 UTC #5 lore raw

PATCH 4/8 Move might_sleep and friends from kernel.h to sched.h

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

These are really related to scheduling, so they should be in sched.h Users usually will need to schedule anyways.

The advantage of having them there is that we can access some of the scheduler inlines to make their fast path more efficient. This will come in a followon patch.

Signed-off-by Andi Kleen <[email protected]>

---
 include/linux/kernel.h | 35 -----------------------------------
 include/linux/sched.h  | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+), 35 deletions(-)

include/linux/kernel.h +0 ~~-35~~

index 482ad2d..badcc13 100644
@@ -141,35 +141,6 @@ struct completion;
 struct pt_regs;
 struct user;
 
-#ifdef CONFIG_PREEMPT_VOLUNTARY
-extern int _cond_resched(void);
-# define might_resched() _cond_resched()
-#else
-# define might_resched() do { } while (0)
-#endif
-
-#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
-  void __might_sleep(const char *file, int line, int preempt_offset);
-/**
- * might_sleep - annotation for functions that can sleep
- *
- * this macro will print a stack trace if it is executed in an atomic
- * context (spinlock, irq-handler, ...).
- *
- * This is a useful debugging help to be able to catch problems early and not
- * be bitten later when the calling function happens to sleep when it is not
- * supposed to.
- */
-# define might_sleep() \
-	do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0)
-#else
-  static inline void __might_sleep(const char *file, int line,
-				   int preempt_offset) { }
-# define might_sleep() do { might_resched(); } while (0)
-#endif
-
-#define might_sleep_if(cond) do { if (cond) might_sleep(); } while (0)
-
 /*
  * abs() handles unsigned and signed longs, ints, shorts and chars.  For all
  * input types abs() returns a signed long.

@@ -193,12 +164,6 @@ extern int _cond_resched(void);
 		(__x < 0) ? -__x : __x;		\
 	})
 
-#if defined(CONFIG_PROVE_LOCKING) || defined(CONFIG_DEBUG_ATOMIC_SLEEP)
-void might_fault(void);
-#else
-static inline void might_fault(void) { }
-#endif
-
 extern struct atomic_notifier_head panic_notifier_list;
 extern long (*panic_blink)(int state);
 __printf(1, 2)

include/linux/sched.h +38 -0

index d722490..773f21d 100644
@@ -2433,6 +2433,35 @@ extern int __cond_resched_softirq(void);
 	__cond_resched_softirq();					\
 })
 
+#ifdef CONFIG_PREEMPT_VOLUNTARY
+extern int _cond_resched(void);
+# define might_resched() _cond_resched()
+#else
+# define might_resched() do { } while (0)
+#endif
+
+#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
+ void __might_sleep(const char *file, int line, int preempt_offset);
+/**
+ * might_sleep - annotation for functions that can sleep
+ *
+ * this macro will print a stack trace if it is executed in an atomic
+ * context (spinlock, irq-handler, ...).
+ *
+ * This is a useful debugging help to be able to catch problems early and not
+ * be bitten later when the calling function happens to sleep when it is not
+ * supposed to.
+ */
+# define might_sleep() \
+	do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0)
+#else
+  static inline void __might_sleep(const char *file, int line,
+				   int preempt_offset) { }
+# define might_sleep() do { might_resched(); } while (0)
+#endif
+
+#define might_sleep_if(cond) do { if (cond) might_sleep(); } while (0)
+
 static inline void cond_resched_rcu(void)
 {
 #if defined(CONFIG_DEBUG_ATOMIC_SLEEP) || !defined(CONFIG_PREEMPT_RCU)

@@ -2442,6 +2471,15 @@ static inline void cond_resched_rcu(void)
 #endif
 }
 
+#ifdef CONFIG_PROVE_LOCKING
+void might_fault(void);
+#else
+static inline void might_fault(void)
+{
+	might_sleep();
+}
+#endif
+
 /*
  * Does a critical section need to be broken due to another
  * task waiting?: (technically does not depend on CONFIG_PREEMPT,

Signature · 1 line

1.8.3.1

Andi Kleen

author

2013-08-14 00:07 UTC #6 lore raw

PATCH 5/8 sched: mark should_resched() __always_inline

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

At least gcc 4.6 and some earlier ones does not inline this function. Since it's small and on relatively hot paths force inline it.

Signed-off-by Andi Kleen <[email protected]>

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/core.c +1 -1

index 74d7c04..23df96a 100644
@@ -3767,7 +3767,7 @@ SYSCALL_DEFINE0(sched_yield)
 	return 0;
 }
 
-static inline int should_resched(void)
+static __always_inline int should_resched(void)
 {
 	return need_resched() && !(preempt_count() & PREEMPT_ACTIVE);
 }

Signature · 1 line

1.8.3.1

Andi Kleen

author

2013-08-14 00:07 UTC #7 lore raw

PATCH 6/8 Add might_fault_debug_only()

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

Add a might_fault_debug_only() that only does something in the PROVE_LOCKING case, but does not cond_resched for PREEMPT_VOLUNTARY. This is for cases when the cond_resched is done elsewhere

Signed-off-by Andi Kleen <[email protected]>

---
 include/linux/sched.h | 2 ++
 1 file changed, 2 insertions(+)

include/linux/sched.h +2 -0

index 773f21d..bb7a08a 100644
@@ -2473,11 +2473,13 @@ static inline void cond_resched_rcu(void)
 
 #ifdef CONFIG_PROVE_LOCKING
 void might_fault(void);
+#define might_fault_debug_only() might_fault()
 #else
 static inline void might_fault(void)
 {
 	might_sleep();
 }
+#define might_fault_debug_only() do {} while(0)
 #endif
 
 /*

Signature · 1 line

1.8.3.1

Andi Kleen

author

2013-08-14 00:07 UTC #8 lore raw

PATCH 7/8 x86: Remove cond_resched() from uaccess code

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

As suggested by Linus, remove cond_resched() from the x86 uaccess code. Now we only do might_fault() in debug kernels.

This means *_user() is not a reschedule point anymore for CONFIG_PREEMPT_VOLUNTARY, only explicit cond_resched()s are.

Even in the debug kernels we should probably move
it out of line where possible, but that's left for future patches.

I did some tests with ftrace's max wakeup latency tracer and CONFIG_PREEMPT_VOLUNTARY:

          no-resched   resched
aim7      45 us        319 us
ebizzy    123 us       117 us
hackbench 416 us       50 us
kbench    14960 us     19741 us

I'm not sure the results are very conclusive, as they go both ways. Most likely it costs a bit.

Signed-off-by Andi Kleen <[email protected]>

---
 arch/x86/include/asm/uaccess.h    |  4 ++--
 arch/x86/include/asm/uaccess_32.h |  6 +++---
 arch/x86/include/asm/uaccess_64.h | 12 ++++++------
 3 files changed, 11 insertions(+), 11 deletions(-)

arch/x86/include/asm/uaccess.h +2 -2

index 8fa3bd6..c860ebe 100644
@@ -165,7 +165,7 @@ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL))
 	int __ret_gu;							\
 	register __inttype(*(ptr)) __val_gu asm("%edx");		\
 	__chk_user_ptr(ptr);						\
-	might_fault();							\
+	might_fault_debug_only();					\
 	asm volatile("call __get_user_%P3"				\
 		     : "=a" (__ret_gu), "=r" (__val_gu)			\
 		     : "0" (ptr), "i" (sizeof(*(ptr))));		\

@@ -246,7 +246,7 @@ extern void __put_user_8(void);
 	int __ret_pu;						\
 	__typeof__(*(ptr)) __pu_val;				\
 	__chk_user_ptr(ptr);					\
-	might_fault();						\
+	might_fault_debug_only();				\
 	__pu_val = x;						\
 	switch (sizeof(*(ptr))) {				\
 	case 1:							\

arch/x86/include/asm/uaccess_32.h +3 -3

index 7f760a9..e656ee9 100644
@@ -81,7 +81,7 @@ __copy_to_user_inatomic(void __user *to, const void *from, unsigned long n)
 static __always_inline unsigned long __must_check
 __copy_to_user(void __user *to, const void *from, unsigned long n)
 {
-	might_fault();
+	might_fault_debug_only();
 	return __copy_to_user_inatomic(to, from, n);
 }

@@ -136,7 +136,7 @@ __copy_from_user_inatomic(void *to, const void __user *from, unsigned long n)
 static __always_inline unsigned long
 __copy_from_user(void *to, const void __user *from, unsigned long n)
 {
-	might_fault();
+	might_fault_debug_only();
 	if (__builtin_constant_p(n)) {
 		unsigned long ret;

@@ -158,7 +158,7 @@ __copy_from_user(void *to, const void __user *from, unsigned long n)
 static __always_inline unsigned long __copy_from_user_nocache(void *to,
 				const void __user *from, unsigned long n)
 {
-	might_fault();
+	might_fault_debug_only();
 	if (__builtin_constant_p(n)) {
 		unsigned long ret;

arch/x86/include/asm/uaccess_64.h +6 -6

index 64476bb..5a3bb4e 100644
@@ -58,7 +58,7 @@ static inline unsigned long __must_check copy_from_user(void *to,
 {
 	int sz = __compiletime_object_size(to);
 
-	might_fault();
+	might_fault_debug_only();
 	if (likely(sz == -1 || sz >= n))
 		n = _copy_from_user(to, from, n);
 #ifdef CONFIG_DEBUG_VM

@@ -71,7 +71,7 @@ static inline unsigned long __must_check copy_from_user(void *to,
 static __always_inline __must_check
 int copy_to_user(void __user *dst, const void *src, unsigned size)
 {
-	might_fault();
+	might_fault_debug_only();
 
 	return _copy_to_user(dst, src, size);
 }

@@ -122,7 +122,7 @@ int __copy_from_user_nocheck(void *dst, const void __user *src, unsigned size)
 static __always_inline __must_check
 int __copy_from_user(void *dst, const void __user *src, unsigned size)
 {
-	might_fault();
+	might_fault_debug_only();
 	return __copy_from_user_nocheck(dst, src, size);
 }

@@ -172,7 +172,7 @@ int __copy_to_user_nocheck(void __user *dst, const void *src, unsigned size)
 static __always_inline __must_check
 int __copy_to_user(void __user *dst, const void *src, unsigned size)
 {
-	might_fault();
+	might_fault_debug_only();
 	return __copy_to_user_nocheck(dst, src, size);
 }

@@ -181,7 +181,7 @@ int __copy_in_user(void __user *dst, const void __user *src, unsigned size)
 {
 	int ret = 0;
 
-	might_fault();
+	might_fault_debug_only();
 	if (!__builtin_constant_p(size))
 		return copy_user_generic((__force void *)dst,
 					 (__force void *)src, size);

@@ -247,7 +247,7 @@ extern long __copy_user_nocache(void *dst, const void __user *src,
 static inline int
 __copy_from_user_nocache(void *dst, const void __user *src, unsigned size)
 {
-	might_fault();
+	might_fault_debug_only();
 	return __copy_user_nocache(dst, src, size, 1);
 }

Signature · 1 line

1.8.3.1

Andi Kleen

author

2013-08-14 00:07 UTC #9 lore raw

PATCH 8/8 sched: Inline the need_resched test into the caller for _cond_resched

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

_cond_resched is very common in kernel calls, e.g. it's used in every user access. Usually it does at least two explicit calls just to decide to do
nothing: _cond_resched and should_resched(). Inline a need_resched() into the caller to avoid these calls in the common case of no reschedule being needed.

Previously this would have been very expensive in terms of binary size because there were a lot of inlined cond_resched()s in copy_*_user() and put/get_user(). But with the newest changes to x86 uaccess.h these not inlined anymore, so we can use a slightly bigger, but much faster fast path version.

Signed-off-by Andi Kleen <[email protected]>

---
 include/linux/sched.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

include/linux/sched.h +1 -1

index bb7a08a..9e0efa9 100644
@@ -2435,7 +2435,7 @@ extern int __cond_resched_softirq(void);
 
 #ifdef CONFIG_PREEMPT_VOLUNTARY
 extern int _cond_resched(void);
-# define might_resched() _cond_resched()
+# define might_resched() (need_resched() ? _cond_resched() : 0)
 #else
 # define might_resched() do { } while (0)
 #endif

Signature · 1 line

1.8.3.1

Ingo Molnar

2013-08-14 09:56 UTC #10 lore raw

In reply to Andi Kleen

Andi,

You _again_ 'forgot' to Cc: peterz who is an affected maintainer and who is keenly interested in such low level changes affecting scheduling - and he asked to be Cc:-ed on your previous submission.

I still don't understand, why do you *routinely* do office politics crap like that, playing games with Cc:s and private mails, which eminently hinders kernel development? (Oh, it's deniable and I'm quite sure you'll deny it in a heartbeat and call it an inadvertent omission. Just skip the excuses and stop it, ok?)

Thanks,

Ingo

Quoted 41 lines · Andi Kleen

The x86 user access functions (*_user) were originally very well tuned, with partial inline code and other optimizations.
Then over time various new checks -- particularly the sleep checks for a voluntary preempt kernel -- destroyed a lot of the tunings
A typical user access operation is now doing multiple useless function calls. Also the without force inline gcc's inlining policy makes it even worse, with adding more unnecessary calls.
Here's a typical example from ftrace:
     10)               |    might_fault() {
     10)               |      _cond_resched() {
     10)               |        should_resched() {
     10)               |          need_resched() {
     10)   0.063 us    |            test_ti_thread_flag();
     10)   0.643 us    |          }
     10)   1.238 us    |        }
     10)   1.845 us    |      }
     10)   2.438 us    |    }
So we spent 2.5us doing nothing (ok it's a bit less without ftrace, but still pretty bad)
Then in other cases we would have an out of line function, but would actually do the might_sleep() checks in the inlined caller. This doesn't make any sense at all.
There were also a few other problems, for example the x86-64 uaccess code regularly falls back to string functions, even though a simple mov would be enough. For example every futex access to the lock variable would actually use string instructions, even though it's just 4 bytes.
This patch kit is an attempt to get us back to sane code, mostly by doing proper inlining and doing sleep checks in the right place. Unfortunately I had to add one tree sweep to avoid an nasty include loop.
v2: Now completely remove reschedule checks for uaccess functions.