Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
Hi
When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.
Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
My HW - t61, 4GB
Zdenek
BUG: unable to handle kernel NULL pointer dereference at 0000000000000178 IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] PGD 0 Oops: 0000 [#1] PREEMPT SMP CPU 1 Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight
Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO
RIP: 0010:[<ffffffffa031204d>] [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP: 0018:ffff880126b77c68 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607 RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40 RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400 R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task ffff8801344ca310) Stack: ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0 Call Trace: [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror] [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror] [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod] [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod] [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod] [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod] [<ffffffff81060065>] process_one_work+0x1a5/0x6b0 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod] [<ffffffff810608ee>] worker_thread+0x12e/0x2d0 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210 [<ffffffff81069cf2>] kthread+0xa2/0xb0 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0 [<ffffffff81533330>] ? gs_change+0xb/0xb Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 RIP [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP <ffff880126b77c68>
CR2: 0000000000000178
# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug fs entries on failure of file creation and made extfrag_debug_root dentry local git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105
CONFIG_64BIT=yCONFIG_X86_64=yCONFIG_X86=yCONFIG_INSTRUCTION_DECODER=yCONFIG_OUTPUT_FORMAT="elf64-x86-64"CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"CONFIG_LOCKDEP_SUPPORT=yCONFIG_STACKTRACE_SUPPORT=yCONFIG_HAVE_LATENCYTOP_SUPPORT=yCONFIG_MMU=yCONFIG_NEED_DMA_MAP_STATE=yCONFIG_NEED_SG_DMA_LENGTH=yCONFIG_GENERIC_ISA_DMA=yCONFIG_GENERIC_BUG=yCONFIG_GENERIC_BUG_RELATIVE_POINTERS=yCONFIG_GENERIC_HWEIGHT=yCONFIG_ARCH_MAY_HAVE_PC_FDC=yCONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_GENERIC_CALIBRATE_DELAY=yCONFIG_ARCH_HAS_CPU_RELAX=yCONFIG_ARCH_HAS_DEFAULT_IDLE=yCONFIG_ARCH_HAS_CACHE_LINE_SIZE=yCONFIG_ARCH_HAS_CPU_AUTOPROBE=yCONFIG_HAVE_SETUP_PER_CPU_AREA=yCONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=yCONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=yCONFIG_ARCH_HIBERNATION_POSSIBLE=yCONFIG_ARCH_SUSPEND_POSSIBLE=yCONFIG_ZONE_DMA32=yCONFIG_AUDIT_ARCH=yCONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=yCONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=yCONFIG_X86_64_SMP=yCONFIG_X86_HT=yCONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" CONFIG_ARCH_CPU_PROBE_RELEASE=yCONFIG_ARCH_SUPPORTS_UPROBES=yCONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"CONFIG_HAVE_IRQ_WORK=yCONFIG_IRQ_WORK=yCONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_EXPERIMENTAL=yCONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""CONFIG_LOCALVERSION=""CONFIG_LOCALVERSION_AUTO=yCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_BZIP2=yCONFIG_HAVE_KERNEL_LZMA=yCONFIG_HAVE_KERNEL_XZ=yCONFIG_HAVE_KERNEL_LZO=yCONFIG_KERNEL_LZMA=yCONFIG_DEFAULT_HOSTNAME="(none)"CONFIG_SWAP=yCONFIG_SYSVIPC=yCONFIG_SYSVIPC_SYSCTL=yCONFIG_POSIX_MQUEUE=yCONFIG_POSIX_MQUEUE_SYSCTL=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_TASKSTATS=yCONFIG_TASK_DELAY_ACCT=yCONFIG_TASK_XACCT=yCONFIG_TASK_IO_ACCOUNTING=yCONFIG_AUDIT=yCONFIG_AUDITSYSCALL=yCONFIG_AUDIT_WATCH=yCONFIG_AUDIT_TREE=yCONFIG_HAVE_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS=yCONFIG_GENERIC_IRQ_PROBE=yCONFIG_GENERIC_IRQ_SHOW=yCONFIG_GENERIC_PENDING_IRQ=yCONFIG_IRQ_FORCED_THREADING=yCONFIG_SPARSE_IRQ=yCONFIG_CLOCKSOURCE_WATCHDOG=yCONFIG_ARCH_CLOCKSOURCE_DATA=yCONFIG_GENERIC_TIME_VSYSCALL=yCONFIG_GENERIC_CLOCKEVENTS=yCONFIG_GENERIC_CLOCKEVENTS_BUILD=yCONFIG_GENERIC_CLOCKEVENTS_BROADCAST=yCONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=yCONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_TICK_ONESHOT=yCONFIG_NO_HZ=yCONFIG_HIGH_RES_TIMERS=y
CONFIG_TREE_PREEMPT_RCU=yCONFIG_PREEMPT_RCU=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16CONFIG_RCU_FAST_NO_HZ=yCONFIG_TREE_RCU_TRACE=yCONFIG_IKCONFIG=mCONFIG_IKCONFIG_PROC=yCONFIG_LOG_BUF_SHIFT=17CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=yCONFIG_CGROUPS=yCONFIG_CGROUP_FREEZER=yCONFIG_CGROUP_DEVICE=yCONFIG_CPUSETS=yCONFIG_PROC_PID_CPUSET=yCONFIG_CGROUP_CPUACCT=yCONFIG_RESOURCE_COUNTERS=yCONFIG_CGROUP_SCHED=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_CFS_BANDWIDTH=yCONFIG_RT_GROUP_SCHED=yCONFIG_BLK_CGROUP=yCONFIG_NAMESPACES=yCONFIG_UTS_NS=yCONFIG_IPC_NS=yCONFIG_PID_NS=yCONFIG_NET_NS=yCONFIG_SCHED_AUTOGROUP=yCONFIG_SYSFS_DEPRECATED=yCONFIG_RELAY=yCONFIG_BLK_DEV_INITRD=yCONFIG_INITRAMFS_SOURCE=""CONFIG_RD_GZIP=yCONFIG_RD_BZIP2=yCONFIG_RD_LZMA=yCONFIG_RD_XZ=yCONFIG_RD_LZO=yCONFIG_SYSCTL=yCONFIG_ANON_INODES=yCONFIG_UID16=yCONFIG_KALLSYMS=yCONFIG_KALLSYMS_ALL=yCONFIG_HOTPLUG=yCONFIG_PRINTK=yCONFIG_BUG=yCONFIG_ELF_CORE=yCONFIG_PCSPKR_PLATFORM=yCONFIG_HAVE_PCSPKR_PLATFORM=yCONFIG_BASE_FULL=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_SIGNALFD=yCONFIG_TIMERFD=yCONFIG_EVENTFD=yCONFIG_SHMEM=yCONFIG_AIO=yCONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_EVENTS=yCONFIG_VM_EVENT_COUNTERS=yCONFIG_PCI_QUIRKS=yCONFIG_SLUB_DEBUG=yCONFIG_SLUB=yCONFIG_PROFILING=yCONFIG_TRACEPOINTS=yCONFIG_OPROFILE=mCONFIG_HAVE_OPROFILE=yCONFIG_OPROFILE_NMI_TIMER=yCONFIG_KPROBES=yCONFIG_JUMP_LABEL=yCONFIG_UPROBES=yCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=yCONFIG_KRETPROBES=yCONFIG_USER_RETURN_NOTIFIER=yCONFIG_HAVE_IOREMAP_PROT=yCONFIG_HAVE_KPROBES=yCONFIG_HAVE_KRETPROBES=yCONFIG_HAVE_OPTPROBES=yCONFIG_HAVE_ARCH_TRACEHOOK=yCONFIG_HAVE_DMA_ATTRS=yCONFIG_USE_GENERIC_SMP_HELPERS=yCONFIG_GENERIC_SMP_IDLE_THREAD=yCONFIG_HAVE_REGS_AND_STACK_ACCESS_API=yCONFIG_HAVE_DMA_API_DEBUG=yCONFIG_HAVE_HW_BREAKPOINT=yCONFIG_HAVE_MIXED_BREAKPOINTS_REGS=yCONFIG_HAVE_USER_RETURN_NOTIFIER=yCONFIG_HAVE_PERF_EVENTS_NMI=yCONFIG_HAVE_ARCH_JUMP_LABEL=yCONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=yCONFIG_HAVE_ALIGNED_STRUCT_PAGE=yCONFIG_HAVE_CMPXCHG_LOCAL=yCONFIG_HAVE_CMPXCHG_DOUBLE=yCONFIG_ARCH_WANT_OLD_COMPAT_IPC=yCONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SLABINFO=yCONFIG_RT_MUTEXES=yCONFIG_BASE_SMALL=0CONFIG_MODULES=yCONFIG_MODULE_UNLOAD=yCONFIG_MODULE_FORCE_UNLOAD=yCONFIG_MODULE_SRCVERSION_ALL=yCONFIG_STOP_MACHINE=yCONFIG_BLOCK=yCONFIG_BLK_DEV_BSG=yCONFIG_BLK_DEV_INTEGRITY=yCONFIG_BLK_DEV_THROTTLING=y
CONFIG_PARTITION_ADVANCED=yCONFIG_AMIGA_PARTITION=yCONFIG_MSDOS_PARTITION=yCONFIG_LDM_PARTITION=yCONFIG_EFI_PARTITION=yCONFIG_BLOCK_COMPAT=y
CONFIG_IOSCHED_NOOP=yCONFIG_IOSCHED_DEADLINE=yCONFIG_IOSCHED_CFQ=mCONFIG_DEFAULT_DEADLINE=yCONFIG_DEFAULT_IOSCHED="deadline"CONFIG_PREEMPT_NOTIFIERS=yCONFIG_PADATA=yCONFIG_UNINLINE_SPIN_UNLOCK=yCONFIG_FREEZER=y
CONFIG_ZONE_DMA=yCONFIG_SMP=yCONFIG_X86_MPPARSE=yCONFIG_X86_SUPPORTS_MEMORY_FAILURE=yCONFIG_SCHED_OMIT_FRAME_POINTER=yCONFIG_NO_BOOTMEM=yCONFIG_MEMTEST=yCONFIG_MCORE2=yCONFIG_X86_INTERNODE_CACHE_SHIFT=6CONFIG_X86_CMPXCHG=yCONFIG_X86_L1_CACHE_SHIFT=6CONFIG_X86_XADD=yCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INTEL_USERCOPY=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_X86_P6_NOP=yCONFIG_X86_TSC=yCONFIG_X86_CMPXCHG64=yCONFIG_X86_CMOV=yCONFIG_X86_MINIMUM_CPU_FAMILY=64CONFIG_X86_DEBUGCTLMSR=yCONFIG_CPU_SUP_INTEL=yCONFIG_CPU_SUP_AMD=yCONFIG_CPU_SUP_CENTAUR=yCONFIG_HPET_TIMER=yCONFIG_HPET_EMULATE_RTC=yCONFIG_DMI=yCONFIG_GART_IOMMU=yCONFIG_SWIOTLB=yCONFIG_IOMMU_HELPER=yCONFIG_NR_CPUS=2CONFIG_PREEMPT=yCONFIG_PREEMPT_COUNT=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=yCONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=yCONFIG_X86_MCE=yCONFIG_X86_MCE_INTEL=yCONFIG_X86_MCE_AMD=yCONFIG_X86_MCE_THRESHOLD=yCONFIG_X86_THERMAL_VECTOR=yCONFIG_MICROCODE=mCONFIG_MICROCODE_INTEL=yCONFIG_MICROCODE_AMD=yCONFIG_MICROCODE_OLD_INTERFACE=yCONFIG_X86_MSR=mCONFIG_X86_CPUID=mCONFIG_ARCH_PHYS_ADDR_T_64BIT=yCONFIG_ARCH_DMA_ADDR_T_64BIT=yCONFIG_DIRECT_GBPAGES=yCONFIG_ARCH_SPARSEMEM_ENABLE=yCONFIG_ARCH_SPARSEMEM_DEFAULT=yCONFIG_ARCH_SELECT_MEMORY_MODEL=yCONFIG_ARCH_PROC_KCORE_TEXT=yCONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000CONFIG_SELECT_MEMORY_MODEL=yCONFIG_SPARSEMEM_MANUAL=yCONFIG_SPARSEMEM=yCONFIG_HAVE_MEMORY_PRESENT=yCONFIG_SPARSEMEM_EXTREME=yCONFIG_SPARSEMEM_VMEMMAP_ENABLE=yCONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=yCONFIG_SPARSEMEM_VMEMMAP=yCONFIG_HAVE_MEMBLOCK=yCONFIG_HAVE_MEMBLOCK_NODE_MAP=yCONFIG_ARCH_DISCARD_MEMBLOCK=yCONFIG_PAGEFLAGS_EXTENDED=yCONFIG_SPLIT_PTLOCK_CPUS=999999CONFIG_COMPACTION=yCONFIG_MIGRATION=yCONFIG_PHYS_ADDR_T_64BIT=yCONFIG_ZONE_DMA_FLAG=1CONFIG_BOUNCE=yCONFIG_VIRT_TO_BUS=yCONFIG_MMU_NOTIFIER=yCONFIG_KSM=yCONFIG_DEFAULT_MMAP_MIN_ADDR=4096CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=yCONFIG_CROSS_MEMORY_ATTACH=yCONFIG_CLEANCACHE=yCONFIG_FRONTSWAP=yCONFIG_X86_CHECK_BIOS_CORRUPTION=yCONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=yCONFIG_X86_RESERVE_LOW=64CONFIG_MTRR=yCONFIG_MTRR_SANITIZER=yCONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1CONFIG_X86_PAT=yCONFIG_ARCH_USES_PG_UNCACHED=yCONFIG_ARCH_RANDOM=yCONFIG_EFI=yCONFIG_HZ_300=yCONFIG_HZ=300CONFIG_SCHED_HRTICK=yCONFIG_KEXEC=yCONFIG_KEXEC_JUMP=yCONFIG_PHYSICAL_START=0x1000000CONFIG_PHYSICAL_ALIGN=0x1000000CONFIG_HOTPLUG_CPU=yCONFIG_CMDLINE_BOOL=yCONFIG_CMDLINE=""CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_HIBERNATION_HEADER=yCONFIG_SUSPEND=yCONFIG_SUSPEND_FREEZER=yCONFIG_HIBERNATE_CALLBACKS=yCONFIG_HIBERNATION=yCONFIG_PM_STD_PARTITION=""CONFIG_PM_SLEEP=yCONFIG_PM_SLEEP_SMP=yCONFIG_PM_WAKELOCKS=yCONFIG_PM_WAKELOCKS_LIMIT=100CONFIG_PM_WAKELOCKS_GC=yCONFIG_PM_RUNTIME=yCONFIG_PM=yCONFIG_PM_DEBUG=yCONFIG_PM_ADVANCED_DEBUG=yCONFIG_CAN_PM_TRACE=yCONFIG_PM_TRACE=yCONFIG_PM_TRACE_RTC=yCONFIG_ACPI=yCONFIG_ACPI_SLEEP=yCONFIG_ACPI_PROCFS=yCONFIG_ACPI_PROCFS_POWER=yCONFIG_ACPI_AC=yCONFIG_ACPI_BATTERY=yCONFIG_ACPI_BUTTON=yCONFIG_ACPI_VIDEO=mCONFIG_ACPI_FAN=yCONFIG_ACPI_DOCK=yCONFIG_ACPI_PROCESSOR=yCONFIG_ACPI_HOTPLUG_CPU=yCONFIG_ACPI_PROCESSOR_AGGREGATOR=mCONFIG_ACPI_THERMAL=yCONFIG_ACPI_BLACKLIST_YEAR=0CONFIG_ACPI_DEBUG=yCONFIG_ACPI_PCI_SLOT=yCONFIG_X86_PM_TIMER=yCONFIG_ACPI_CONTAINER=yCONFIG_ACPI_CUSTOM_METHOD=m
CONFIG_CPU_FREQ=yCONFIG_CPU_FREQ_TABLE=yCONFIG_CPU_FREQ_STAT=yCONFIG_CPU_FREQ_STAT_DETAILS=yCONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=yCONFIG_CPU_FREQ_GOV_PERFORMANCE=yCONFIG_CPU_FREQ_GOV_POWERSAVE=yCONFIG_CPU_FREQ_GOV_USERSPACE=yCONFIG_CPU_FREQ_GOV_ONDEMAND=yCONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_X86_PCC_CPUFREQ=mCONFIG_X86_ACPI_CPUFREQ=yCONFIG_X86_P4_CLOCKMOD=m
CONFIG_X86_SPEEDSTEP_LIB=mCONFIG_CPU_IDLE=yCONFIG_CPU_IDLE_GOV_LADDER=yCONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_PCI=yCONFIG_PCI_DIRECT=yCONFIG_PCI_MMCONFIG=yCONFIG_PCI_DOMAINS=yCONFIG_PCIEPORTBUS=yCONFIG_PCIEAER=yCONFIG_PCIEASPM=yCONFIG_PCIEASPM_DEFAULT=yCONFIG_PCIE_PME=yCONFIG_ARCH_SUPPORTS_MSI=yCONFIG_PCI_MSI=yCONFIG_PCI_STUB=yCONFIG_HT_IRQ=yCONFIG_PCI_ATS=yCONFIG_PCI_IOV=yCONFIG_PCI_PRI=yCONFIG_PCI_PASID=yCONFIG_PCI_IOAPIC=yCONFIG_PCI_LABEL=yCONFIG_ISA_DMA_API=yCONFIG_AMD_NB=yCONFIG_PCCARD=yCONFIG_PCMCIA=mCONFIG_PCMCIA_LOAD_CIS=yCONFIG_CARDBUS=y
CONFIG_YENTA=mCONFIG_YENTA_O2=yCONFIG_YENTA_RICOH=yCONFIG_YENTA_TI=yCONFIG_YENTA_ENE_TUNE=yCONFIG_YENTA_TOSHIBA=yCONFIG_PCCARD_NONSTATIC=yCONFIG_HOTPLUG_PCI=y
CONFIG_BINFMT_ELF=yCONFIG_COMPAT_BINFMT_ELF=yCONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=yCONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=yCONFIG_BINFMT_MISC=mCONFIG_IA32_EMULATION=yCONFIG_IA32_AOUT=mCONFIG_X86_X32=yCONFIG_COMPAT=yCONFIG_COMPAT_FOR_U64_ALIGNMENT=yCONFIG_SYSVIPC_COMPAT=yCONFIG_KEYS_COMPAT=yCONFIG_HAVE_TEXT_POKE_SMP=yCONFIG_X86_DEV_DMA_OPS=yCONFIG_NET=yCONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_PACKET=yCONFIG_UNIX=yCONFIG_UNIX_DIAG=mCONFIG_XFRM=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_PNP=yCONFIG_IP_PNP_DHCP=yCONFIG_IP_PNP_BOOTP=yCONFIG_IP_MROUTE=yCONFIG_SYN_COOKIES=yCONFIG_INET_TUNNEL=mCONFIG_TCP_CONG_ADVANCED=yCONFIG_TCP_CONG_BIC=mCONFIG_TCP_CONG_CUBIC=yCONFIG_TCP_CONG_WESTWOOD=mCONFIG_TCP_CONG_HTCP=mCONFIG_DEFAULT_CUBIC=yCONFIG_DEFAULT_TCP_CONG="cubic"CONFIG_IPV6=mCONFIG_INET6_XFRM_MODE_TRANSPORT=mCONFIG_INET6_XFRM_MODE_TUNNEL=mCONFIG_INET6_XFRM_MODE_BEET=mCONFIG_IPV6_SIT=mCONFIG_IPV6_NDISC_NODETYPE=yCONFIG_NETWORK_SECMARK=yCONFIG_NETFILTER=yCONFIG_NETFILTER_ADVANCED=yCONFIG_BRIDGE_NETFILTER=y
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_NETLINK_QUEUE=mCONFIG_NETFILTER_NETLINK_LOG=mCONFIG_NF_CONNTRACK=mCONFIG_NF_CONNTRACK_MARK=yCONFIG_NF_CONNTRACK_SECMARK=yCONFIG_NF_CONNTRACK_PROCFS=yCONFIG_NF_CONNTRACK_EVENTS=yCONFIG_NF_CONNTRACK_FTP=mCONFIG_NF_CT_NETLINK=mCONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_MARK=mCONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=mCONFIG_NETFILTER_XT_TARGET_CONNMARK=mCONFIG_NETFILTER_XT_TARGET_CONNSECMARK=mCONFIG_NETFILTER_XT_TARGET_DSCP=mCONFIG_NETFILTER_XT_TARGET_HL=mCONFIG_NETFILTER_XT_TARGET_LED=mCONFIG_NETFILTER_XT_TARGET_MARK=mCONFIG_NETFILTER_XT_TARGET_NFLOG=mCONFIG_NETFILTER_XT_TARGET_NFQUEUE=mCONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=mCONFIG_NETFILTER_XT_MATCH_CONNLIMIT=mCONFIG_NETFILTER_XT_MATCH_CONNMARK=mCONFIG_NETFILTER_XT_MATCH_CONNTRACK=mCONFIG_NETFILTER_XT_MATCH_DSCP=mCONFIG_NETFILTER_XT_MATCH_ECN=mCONFIG_NETFILTER_XT_MATCH_HL=mCONFIG_NETFILTER_XT_MATCH_IPRANGE=mCONFIG_NETFILTER_XT_MATCH_MARK=mCONFIG_NETFILTER_XT_MATCH_MULTIPORT=mCONFIG_NETFILTER_XT_MATCH_OSF=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NETFILTER_XT_MATCH_POLICY=mCONFIG_NETFILTER_XT_MATCH_PHYSDEV=mCONFIG_NETFILTER_XT_MATCH_PKTTYPE=mCONFIG_NETFILTER_XT_MATCH_QUOTA=mCONFIG_NETFILTER_XT_MATCH_SCTP=mCONFIG_NETFILTER_XT_MATCH_STATE=mCONFIG_NETFILTER_XT_MATCH_STATISTIC=mCONFIG_NETFILTER_XT_MATCH_STRING=mCONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NF_DEFRAG_IPV4=mCONFIG_NF_CONNTRACK_IPV4=mCONFIG_NF_CONNTRACK_PROC_COMPAT=yCONFIG_IP_NF_IPTABLES=mCONFIG_IP_NF_MATCH_AH=mCONFIG_IP_NF_MATCH_ECN=mCONFIG_IP_NF_MATCH_TTL=mCONFIG_IP_NF_FILTER=mCONFIG_IP_NF_TARGET_REJECT=mCONFIG_IP_NF_TARGET_ULOG=mCONFIG_NF_NAT=mCONFIG_NF_NAT_NEEDED=yCONFIG_IP_NF_TARGET_MASQUERADE=mCONFIG_IP_NF_TARGET_REDIRECT=mCONFIG_NF_NAT_FTP=mCONFIG_IP_NF_MANGLE=mCONFIG_IP_NF_TARGET_ECN=mCONFIG_IP_NF_TARGET_TTL=mCONFIG_IP_NF_ARPTABLES=mCONFIG_IP_NF_ARPFILTER=mCONFIG_IP_NF_ARP_MANGLE=m
CONFIG_NF_DEFRAG_IPV6=mCONFIG_NF_CONNTRACK_IPV6=mCONFIG_IP6_NF_IPTABLES=mCONFIG_BRIDGE_NF_EBTABLES=mCONFIG_BRIDGE_EBT_T_NAT=mCONFIG_IP_SCTP=mCONFIG_SCTP_HMAC_NONE=yCONFIG_STP=mCONFIG_BRIDGE=mCONFIG_BRIDGE_IGMP_SNOOPING=yCONFIG_LLC=mCONFIG_DNS_RESOLVER=yCONFIG_RPS=yCONFIG_RFS_ACCEL=yCONFIG_XPS=yCONFIG_NETPRIO_CGROUP=mCONFIG_BQL=y
CONFIG_NET_PKTGEN=mCONFIG_IRDA=m
CONFIG_IRTTY_SIR=m
CONFIG_BT=mCONFIG_BT_RFCOMM=mCONFIG_BT_RFCOMM_TTY=yCONFIG_BT_BNEP=mCONFIG_BT_HIDP=m
CONFIG_BT_HCIBTUSB=mCONFIG_WIRELESS=yCONFIG_WEXT_CORE=yCONFIG_WEXT_PROC=yCONFIG_CFG80211=mCONFIG_CFG80211_DEVELOPER_WARNINGS=yCONFIG_CFG80211_DEFAULT_PS=yCONFIG_CFG80211_DEBUGFS=yCONFIG_CFG80211_WEXT=yCONFIG_WIRELESS_EXT_SYSFS=yCONFIG_LIB80211=mCONFIG_MAC80211=mCONFIG_MAC80211_HAS_RC=yCONFIG_MAC80211_RC_MINSTREL=yCONFIG_MAC80211_RC_MINSTREL_HT=yCONFIG_MAC80211_RC_DEFAULT_MINSTREL=yCONFIG_MAC80211_RC_DEFAULT="minstrel_ht"CONFIG_MAC80211_LEDS=yCONFIG_MAC80211_DEBUGFS=yCONFIG_RFKILL=yCONFIG_RFKILL_LEDS=yCONFIG_RFKILL_INPUT=yCONFIG_RFKILL_REGULATOR=mCONFIG_HAVE_BPF_JIT=y
CONFIG_UEVENT_HELPER_PATH=""CONFIG_DEVTMPFS=yCONFIG_DEVTMPFS_MOUNT=yCONFIG_STANDALONE=yCONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_FW_LOADER=yCONFIG_FIRMWARE_IN_KERNEL=yCONFIG_EXTRA_FIRMWARE=""CONFIG_DMA_SHARED_BUFFER=yCONFIG_CONNECTOR=mCONFIG_MTD=mCONFIG_MTD_REDBOOT_PARTS=mCONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
CONFIG_MTD_CHAR=mCONFIG_MTD_BLKDEVS=mCONFIG_MTD_BLOCK=mCONFIG_SM_FTL=m
CONFIG_MTD_MAP_BANK_WIDTH_1=yCONFIG_MTD_MAP_BANK_WIDTH_2=yCONFIG_MTD_MAP_BANK_WIDTH_4=yCONFIG_MTD_CFI_I1=yCONFIG_MTD_CFI_I2=y
CONFIG_MTD_MTDRAM=mCONFIG_MTDRAM_TOTAL_SIZE=4096CONFIG_MTDRAM_ERASE_SIZE=128CONFIG_MTD_BLOCK2MTD=m
CONFIG_MTD_NAND_ECC=mCONFIG_MTD_NAND_ECC_SMC=y
CONFIG_PNP=y
CONFIG_PNPACPI=yCONFIG_BLK_DEV=yCONFIG_BLK_DEV_FD=mCONFIG_BLK_DEV_LOOP=mCONFIG_BLK_DEV_LOOP_MIN_COUNT=8CONFIG_BLK_DEV_CRYPTOLOOP=mCONFIG_BLK_DEV_DRBD=mCONFIG_BLK_DEV_NBD=mCONFIG_BLK_DEV_RAM=yCONFIG_BLK_DEV_RAM_COUNT=16CONFIG_BLK_DEV_RAM_SIZE=8192CONFIG_CDROM_PKTCDVD=mCONFIG_CDROM_PKTCDVD_BUFFERS=8CONFIG_VIRTIO_BLK=y
CONFIG_INTEL_MEI=mCONFIG_HAVE_IDE=y
CONFIG_SCSI_MOD=yCONFIG_RAID_ATTRS=mCONFIG_SCSI=yCONFIG_SCSI_DMA=y
CONFIG_BLK_DEV_SD=yCONFIG_BLK_DEV_SR=mCONFIG_BLK_DEV_SR_VENDOR=yCONFIG_CHR_DEV_SG=yCONFIG_CHR_DEV_SCH=mCONFIG_SCSI_MULTI_LUN=yCONFIG_SCSI_CONSTANTS=yCONFIG_SCSI_LOGGING=yCONFIG_SCSI_SCAN_ASYNC=yCONFIG_SCSI_WAIT_SCAN=m
CONFIG_SCSI_SPI_ATTRS=yCONFIG_SCSI_LOWLEVEL=yCONFIG_SCSI_DEBUG=mCONFIG_SCSI_VIRTIO=mCONFIG_ATA=yCONFIG_ATA_VERBOSE_ERROR=yCONFIG_ATA_ACPI=yCONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=yCONFIG_ATA_SFF=y
CONFIG_ATA_BMDMA=y
CONFIG_ATA_PIIX=y
CONFIG_PATA_MPIIX=y
CONFIG_MD=yCONFIG_BLK_DEV_MD=mCONFIG_MD_LINEAR=mCONFIG_MD_RAID0=mCONFIG_MD_RAID1=mCONFIG_MD_RAID10=mCONFIG_MD_RAID456=mCONFIG_MULTICORE_RAID456=yCONFIG_MD_MULTIPATH=mCONFIG_MD_FAULTY=mCONFIG_BLK_DEV_DM=mCONFIG_DM_DEBUG=yCONFIG_DM_BUFIO=mCONFIG_DM_PERSISTENT_DATA=mCONFIG_DM_CRYPT=mCONFIG_DM_SNAPSHOT=mCONFIG_DM_THIN_PROVISIONING=mCONFIG_DM_MIRROR=mCONFIG_DM_RAID=mCONFIG_DM_LOG_USERSPACE=mCONFIG_DM_ZERO=mCONFIG_DM_MULTIPATH=mCONFIG_DM_DELAY=mCONFIG_DM_UEVENT=yCONFIG_DM_FLAKEY=mCONFIG_DM_VERITY=m
CONFIG_NETDEVICES=yCONFIG_NET_CORE=yCONFIG_DUMMY=mCONFIG_MII=mCONFIG_NETCONSOLE=mCONFIG_NETPOLL=yCONFIG_NETPOLL_TRAP=yCONFIG_NET_POLL_CONTROLLER=yCONFIG_TUN=mCONFIG_VETH=mCONFIG_VIRTIO_NET=m
CONFIG_ETHERNET=yCONFIG_NET_VENDOR_INTEL=yCONFIG_E1000E=mCONFIG_IXGBEVF=mCONFIG_NET_VENDOR_REALTEK=yCONFIG_8139TOO=mCONFIG_8139TOO_PIO=yCONFIG_PPP=mCONFIG_PPP_BSDCOMP=mCONFIG_PPP_DEFLATE=mCONFIG_PPP_FILTER=yCONFIG_PPP_MPPE=mCONFIG_PPP_MULTILINK=yCONFIG_PPP_ASYNC=mCONFIG_SLHC=m
CONFIG_WLAN=yCONFIG_IWLWIFI=m
CONFIG_IWLEGACY=mCONFIG_IWL3945=m
CONFIG_IWLEGACY_DEBUGFS=y
CONFIG_INPUT=yCONFIG_INPUT_POLLDEV=yCONFIG_INPUT_SPARSEKMAP=m
CONFIG_INPUT_MOUSEDEV=yCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768CONFIG_INPUT_EVDEV=m
CONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ATKBD=yCONFIG_INPUT_MOUSE=yCONFIG_MOUSE_PS2=mCONFIG_MOUSE_PS2_ALPS=yCONFIG_MOUSE_PS2_LOGIPS2PP=yCONFIG_MOUSE_PS2_SYNAPTICS=yCONFIG_MOUSE_PS2_LIFEBOOK=yCONFIG_MOUSE_PS2_TRACKPOINT=yCONFIG_MOUSE_SERIAL=mCONFIG_INPUT_MISC=yCONFIG_INPUT_UINPUT=m
CONFIG_SERIO=yCONFIG_SERIO_I8042=yCONFIG_SERIO_SERPORT=mCONFIG_SERIO_PCIPS2=mCONFIG_SERIO_LIBPS2=yCONFIG_SERIO_RAW=m
CONFIG_VT=yCONFIG_CONSOLE_TRANSLATIONS=yCONFIG_VT_CONSOLE=yCONFIG_VT_CONSOLE_SLEEP=yCONFIG_HW_CONSOLE=yCONFIG_UNIX98_PTYS=yCONFIG_DEVKMEM=y
CONFIG_SERIAL_8250=yCONFIG_SERIAL_8250_CONSOLE=yCONFIG_FIX_EARLYCON_MEM=yCONFIG_SERIAL_8250_PCI=yCONFIG_SERIAL_8250_PNP=yCONFIG_SERIAL_8250_NR_UARTS=16CONFIG_SERIAL_8250_RUNTIME_UARTS=4CONFIG_SERIAL_8250_EXTENDED=yCONFIG_SERIAL_8250_SHARE_IRQ=yCONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_CORE=yCONFIG_SERIAL_CORE_CONSOLE=yCONFIG_HVC_DRIVER=yCONFIG_VIRTIO_CONSOLE=mCONFIG_HW_RANDOM=mCONFIG_HW_RANDOM_INTEL=mCONFIG_HW_RANDOM_VIRTIO=mCONFIG_NVRAM=m
CONFIG_HPET=yCONFIG_HPET_MMAP=yCONFIG_HANGCHECK_TIMER=mCONFIG_DEVPORT=yCONFIG_I2C=mCONFIG_I2C_BOARDINFO=yCONFIG_I2C_COMPAT=yCONFIG_I2C_CHARDEV=mCONFIG_I2C_HELPER_AUTO=yCONFIG_I2C_ALGOBIT=m
CONFIG_I2C_I801=mCONFIG_I2C_ISCH=mCONFIG_I2C_PIIX4=m
CONFIG_I2C_TINY_USB=m
CONFIG_I2C_DEBUG_ALGO=yCONFIG_HSI=mCONFIG_HSI_BOARDINFO=y
CONFIG_HSI_CHAR=m
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=yCONFIG_POWER_SUPPLY=yCONFIG_HWMON=y
CONFIG_SENSORS_ACPI_POWER=mCONFIG_THERMAL=yCONFIG_THERMAL_HWMON=yCONFIG_WATCHDOG=yCONFIG_WATCHDOG_CORE=y
CONFIG_SOFT_WATCHDOG=mCONFIG_ITCO_WDT=mCONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_SSB_POSSIBLE=y
CONFIG_BCMA_POSSIBLE=y
CONFIG_MFD_CORE=mCONFIG_LPC_SCH=mCONFIG_LPC_ICH=mCONFIG_REGULATOR=yCONFIG_REGULATOR_DEBUG=yCONFIG_REGULATOR_DUMMY=yCONFIG_REGULATOR_VIRTUAL_CONSUMER=mCONFIG_MEDIA_SUPPORT=m
CONFIG_VIDEO_DEV=mCONFIG_VIDEO_V4L2_COMMON=mCONFIG_DVB_CORE=mCONFIG_VIDEO_MEDIA=m
CONFIG_RC_CORE=mCONFIG_LIRC=mCONFIG_RC_MAP=mCONFIG_IR_SANYO_DECODER=mCONFIG_RC_LOOPBACK=mCONFIG_MEDIA_ATTACH=yCONFIG_MEDIA_TUNER=mCONFIG_MEDIA_TUNER_CUSTOMISE=y
CONFIG_MEDIA_TUNER_SIMPLE=mCONFIG_MEDIA_TUNER_TDA9887=mCONFIG_MEDIA_TUNER_MT2060=mCONFIG_MEDIA_TUNER_MT2266=mCONFIG_MEDIA_TUNER_TDA18218=mCONFIG_MEDIA_TUNER_TDA18212=mCONFIG_VIDEO_V4L2=mCONFIG_VIDEOBUF_GEN=mCONFIG_VIDEOBUF_DMA_SG=mCONFIG_VIDEO_TVEEPROM=mCONFIG_VIDEO_TUNER=mCONFIG_V4L2_MEM2MEM_DEV=mCONFIG_VIDEOBUF2_CORE=mCONFIG_VIDEOBUF2_MEMOPS=mCONFIG_VIDEOBUF2_VMALLOC=mCONFIG_VIDEO_CAPTURE_DRIVERS=yCONFIG_VIDEO_ADV_DEBUG=yCONFIG_VIDEO_HELPER_CHIPS_AUTO=yCONFIG_VIDEO_IR_I2C=m
CONFIG_VIDEO_MSP3400=mCONFIG_VIDEO_CS53L32A=mCONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_SAA6588=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_CX25840=m
CONFIG_VIDEO_CX2341X=m
CONFIG_VIDEO_VIVI=mCONFIG_V4L_USB_DRIVERS=yCONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=yCONFIG_VIDEO_PVRUSB2=mCONFIG_VIDEO_PVRUSB2_SYSFS=yCONFIG_VIDEO_PVRUSB2_DVB=yCONFIG_V4L_PCI_DRIVERS=yCONFIG_VIDEO_SAA7134=mCONFIG_VIDEO_SAA7134_ALSA=mCONFIG_VIDEO_SAA7134_RC=yCONFIG_V4L_MEM2MEM_DRIVERS=yCONFIG_VIDEO_MEM2MEM_TESTDEV=mCONFIG_DVB_MAX_ADAPTERS=8CONFIG_DVB_DYNAMIC_MINORS=yCONFIG_DVB_CAPTURE_DRIVERS=y
CONFIG_DVB_USB=mCONFIG_DVB_USB_DIB0700=mCONFIG_DVB_USB_CXUSB=mCONFIG_DVB_USB_NOVA_T_USB2=mCONFIG_DVB_USB_AF9015=mCONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_FE_CUSTOMISE=y
CONFIG_DVB_TDA8261=mCONFIG_DVB_TDA10071=m
CONFIG_DVB_DIB3000MC=mCONFIG_DVB_DIB7000M=mCONFIG_DVB_DIB7000P=mCONFIG_DVB_AF9013=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_PLL=mCONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_DUMMY_FE=m
CONFIG_AGP=yCONFIG_AGP_AMD64=yCONFIG_AGP_INTEL=yCONFIG_VGA_ARB=yCONFIG_VGA_ARB_MAX_GPUS=16CONFIG_DRM=mCONFIG_DRM_KMS_HELPER=mCONFIG_DRM_TTM=m
CONFIG_DRM_I915=mCONFIG_DRM_MGA=mCONFIG_DRM_CIRRUS_QEMU=mCONFIG_VIDEO_OUTPUT_CONTROL=mCONFIG_FB=yCONFIG_FIRMWARE_EDID=yCONFIG_FB_BOOT_VESA_SUPPORT=yCONFIG_FB_CFB_FILLRECT=yCONFIG_FB_CFB_COPYAREA=yCONFIG_FB_CFB_IMAGEBLIT=yCONFIG_FB_SYS_FILLRECT=mCONFIG_FB_SYS_COPYAREA=mCONFIG_FB_SYS_IMAGEBLIT=mCONFIG_FB_MODE_HELPERS=yCONFIG_FB_TILEBLITTING=y
CONFIG_FB_VESA=yCONFIG_FB_EFI=yCONFIG_BACKLIGHT_LCD_SUPPORT=yCONFIG_LCD_CLASS_DEVICE=mCONFIG_LCD_PLATFORM=mCONFIG_BACKLIGHT_CLASS_DEVICE=mCONFIG_BACKLIGHT_GENERIC=m
CONFIG_VGA_CONSOLE=yCONFIG_VGACON_SOFT_SCROLLBACK=yCONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64CONFIG_DUMMY_CONSOLE=yCONFIG_FRAMEBUFFER_CONSOLE=yCONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=yCONFIG_LOGO=yCONFIG_LOGO_LINUX_CLUT224=yCONFIG_SOUND=mCONFIG_SOUND_OSS_CORE=yCONFIG_SOUND_OSS_CORE_PRECLAIM=yCONFIG_SND=mCONFIG_SND_TIMER=mCONFIG_SND_PCM=mCONFIG_SND_HWDEP=mCONFIG_SND_RAWMIDI=mCONFIG_SND_JACK=yCONFIG_SND_SEQUENCER=mCONFIG_SND_OSSEMUL=yCONFIG_SND_MIXER_OSS=mCONFIG_SND_PCM_OSS=mCONFIG_SND_PCM_OSS_PLUGINS=yCONFIG_SND_SEQUENCER_OSS=yCONFIG_SND_HRTIMER=mCONFIG_SND_SEQ_HRTIMER_DEFAULT=yCONFIG_SND_DYNAMIC_MINORS=yCONFIG_SND_VERBOSE_PROCFS=yCONFIG_SND_VMASTER=yCONFIG_SND_KCTL_JACK=yCONFIG_SND_DMA_SGBUF=yCONFIG_SND_RAWMIDI_SEQ=mCONFIG_SND_EMU10K1_SEQ=mCONFIG_SND_MPU401_UART=mCONFIG_SND_AC97_CODEC=mCONFIG_SND_DRIVERS=yCONFIG_SND_DUMMY=mCONFIG_SND_SERIAL_U16550=mCONFIG_SND_MPU401=mCONFIG_SND_AC97_POWER_SAVE=yCONFIG_SND_AC97_POWER_SAVE_DEFAULT=0CONFIG_SND_PCI=yCONFIG_SND_EMU10K1=mCONFIG_SND_HDA_INTEL=mCONFIG_SND_HDA_PREALLOC_SIZE=64CONFIG_SND_HDA_INPUT_JACK=yCONFIG_SND_HDA_CODEC_REALTEK=yCONFIG_SND_HDA_CODEC_ANALOG=yCONFIG_SND_HDA_CODEC_SIGMATEL=yCONFIG_SND_HDA_CODEC_VIA=yCONFIG_SND_HDA_CODEC_HDMI=yCONFIG_SND_HDA_CODEC_CIRRUS=yCONFIG_SND_HDA_CODEC_CONEXANT=yCONFIG_SND_HDA_CODEC_CMEDIA=yCONFIG_SND_HDA_CODEC_SI3054=yCONFIG_SND_HDA_GENERIC=yCONFIG_SND_HDA_POWER_SAVE=yCONFIG_SND_HDA_POWER_SAVE_DEFAULT=500CONFIG_SND_INTEL8X0=mCONFIG_AC97_BUS=m
CONFIG_HID=mCONFIG_HID_GENERIC=m
CONFIG_HID_A4TECH=mCONFIG_HID_APPLE=mCONFIG_HID_BELKIN=mCONFIG_HID_CHERRY=mCONFIG_HID_CHICONY=mCONFIG_HID_CYPRESS=mCONFIG_HID_EZKEY=mCONFIG_HID_KYE=mCONFIG_HID_KENSINGTON=mCONFIG_HID_LOGITECH=mCONFIG_HID_MICROSOFT=mCONFIG_HID_MONTEREY=m
CONFIG_USB_HID=mCONFIG_USB_HIDDEV=yCONFIG_USB_ARCH_HAS_OHCI=yCONFIG_USB_ARCH_HAS_EHCI=yCONFIG_USB_ARCH_HAS_XHCI=yCONFIG_USB_SUPPORT=yCONFIG_USB_COMMON=mCONFIG_USB_ARCH_HAS_HCD=yCONFIG_USB=mCONFIG_USB_ANNOUNCE_NEW_DEVICES=y
CONFIG_USB_SUSPEND=yCONFIG_USB_MON=m
CONFIG_USB_XHCI_HCD=mCONFIG_USB_EHCI_HCD=mCONFIG_USB_EHCI_ROOT_HUB_TT=yCONFIG_USB_EHCI_TT_NEWSCHED=yCONFIG_USB_OHCI_HCD=mCONFIG_USB_OHCI_LITTLE_ENDIAN=yCONFIG_USB_UHCI_HCD=m
CONFIG_USB_ACM=mCONFIG_USB_PRINTER=mCONFIG_USB_WDM=mCONFIG_USB_TMC=m
CONFIG_USB_STORAGE=mCONFIG_USB_UAS=m
CONFIG_USB_SERIAL=mCONFIG_USB_SERIAL_GENERIC=yCONFIG_USB_SERIAL_IR=mCONFIG_USB_SERIAL_WWAN=mCONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_TEST=m
CONFIG_MMC=m
CONFIG_MMC_BLOCK=mCONFIG_MMC_BLOCK_MINORS=8CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_MMC_SDHCI=mCONFIG_MMC_SDHCI_PCI=mCONFIG_MMC_SDHCI_PLTFM=mCONFIG_MMC_SDRICOH_CS=mCONFIG_NEW_LEDS=yCONFIG_LEDS_CLASS=y
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=mCONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_EDAC=y
CONFIG_EDAC_DEBUG=yCONFIG_EDAC_DECODE_MCE=yCONFIG_EDAC_MM_EDAC=mCONFIG_EDAC_E752X=mCONFIG_RTC_LIB=yCONFIG_RTC_CLASS=yCONFIG_RTC_HCTOSYS=yCONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_INTF_SYSFS=yCONFIG_RTC_INTF_PROC=yCONFIG_RTC_INTF_DEV=y
CONFIG_RTC_DRV_CMOS=y
CONFIG_DMADEVICES=y
CONFIG_INTEL_IOATDMA=mCONFIG_DMA_ENGINE=y
CONFIG_NET_DMA=yCONFIG_DCA=mCONFIG_VIRTIO=yCONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=yCONFIG_VIRTIO_BALLOON=yCONFIG_VIRTIO_MMIO=mCONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_STAGING=yCONFIG_ZRAM=mCONFIG_ZRAM_DEBUG=yCONFIG_ZCACHE=yCONFIG_ZSMALLOC=y
CONFIG_USB_WPAN_HCD=mCONFIG_X86_PLATFORM_DEVICES=yCONFIG_THINKPAD_ACPI=mCONFIG_THINKPAD_ACPI_ALSA_SUPPORT=yCONFIG_THINKPAD_ACPI_DEBUG=yCONFIG_THINKPAD_ACPI_VIDEO=yCONFIG_SENSORS_HDAPS=yCONFIG_ACPI_WMI=mCONFIG_ACPI_TOSHIBA=mCONFIG_INTEL_IPS=m
CONFIG_CLKEVT_I8253=yCONFIG_I8253_LOCK=yCONFIG_CLKBLD_I8253=yCONFIG_IOMMU_SUPPORT=y
CONFIG_VIRT_DRIVERS=yCONFIG_PM_DEVFREQ=y
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=yCONFIG_DEVFREQ_GOV_PERFORMANCE=yCONFIG_DEVFREQ_GOV_POWERSAVE=yCONFIG_DEVFREQ_GOV_USERSPACE=y
CONFIG_EDD=mCONFIG_FIRMWARE_MEMMAP=yCONFIG_EFI_VARS=mCONFIG_DMIID=yCONFIG_DMI_SYSFS=m
CONFIG_DCACHE_WORD_ACCESS=yCONFIG_EXT2_FS=yCONFIG_EXT2_FS_XATTR=yCONFIG_EXT2_FS_POSIX_ACL=yCONFIG_EXT2_FS_SECURITY=yCONFIG_EXT3_FS=yCONFIG_EXT3_DEFAULTS_TO_ORDERED=yCONFIG_EXT3_FS_XATTR=yCONFIG_EXT3_FS_POSIX_ACL=yCONFIG_EXT3_FS_SECURITY=yCONFIG_EXT4_FS=yCONFIG_EXT4_FS_XATTR=yCONFIG_EXT4_FS_POSIX_ACL=yCONFIG_EXT4_FS_SECURITY=yCONFIG_JBD=yCONFIG_JBD2=yCONFIG_FS_MBCACHE=yCONFIG_REISERFS_FS=mCONFIG_REISERFS_PROC_INFO=yCONFIG_REISERFS_FS_XATTR=yCONFIG_REISERFS_FS_POSIX_ACL=yCONFIG_REISERFS_FS_SECURITY=yCONFIG_XFS_FS=mCONFIG_XFS_QUOTA=yCONFIG_XFS_POSIX_ACL=yCONFIG_XFS_RT=yCONFIG_BTRFS_FS=mCONFIG_BTRFS_FS_POSIX_ACL=yCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_FS_POSIX_ACL=yCONFIG_EXPORTFS=mCONFIG_FILE_LOCKING=yCONFIG_FSNOTIFY=yCONFIG_DNOTIFY=yCONFIG_INOTIFY_USER=yCONFIG_FANOTIFY=yCONFIG_FANOTIFY_ACCESS_PERMISSIONS=yCONFIG_QUOTACTL=yCONFIG_QUOTACTL_COMPAT=yCONFIG_AUTOFS4_FS=mCONFIG_FUSE_FS=mCONFIG_CUSE=mCONFIG_GENERIC_ACL=y
CONFIG_FSCACHE=mCONFIG_FSCACHE_STATS=yCONFIG_FSCACHE_HISTOGRAM=yCONFIG_FSCACHE_DEBUG=yCONFIG_CACHEFILES=mCONFIG_CACHEFILES_DEBUG=y
CONFIG_ISO9660_FS=mCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_UDF_FS=mCONFIG_UDF_NLS=y
CONFIG_FAT_FS=mCONFIG_VFAT_FS=mCONFIG_FAT_DEFAULT_CODEPAGE=852CONFIG_FAT_DEFAULT_IOCHARSET="cp1250"CONFIG_NTFS_FS=mCONFIG_NTFS_RW=y
CONFIG_PROC_FS=yCONFIG_PROC_KCORE=yCONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=yCONFIG_SYSFS=yCONFIG_TMPFS=yCONFIG_TMPFS_POSIX_ACL=yCONFIG_TMPFS_XATTR=yCONFIG_HUGETLBFS=yCONFIG_HUGETLB_PAGE=yCONFIG_CONFIGFS_FS=m CONFIG_MISC_FILESYSTEMS=yCONFIG_JFFS2_FS=mCONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=yCONFIG_JFFS2_LZO=yCONFIG_JFFS2_RTIME=yCONFIG_JFFS2_RUBIN=y CONFIG_JFFS2_CMODE_PRIORITY=yCONFIG_CRAMFS=mCONFIG_SQUASHFS=mCONFIG_SQUASHFS_ZLIB=yCONFIG_SQUASHFS_XZ=y CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=mCONFIG_NFS_V2=yCONFIG_NFS_V3=yCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=yCONFIG_NFS_V4_1=yCONFIG_PNFS_FILE_LAYOUT=mCONFIG_PNFS_BLOCK=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" CONFIG_NFS_USE_KERNEL_DNS=yCONFIG_NFSD=mCONFIG_NFSD_V2_ACL=yCONFIG_NFSD_V3=yCONFIG_NFSD_V3_ACL=yCONFIG_NFSD_V4=yCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=mCONFIG_NFS_COMMON=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=yCONFIG_CIFS=mCONFIG_CIFS_XATTR=yCONFIG_CIFS_POSIX=yCONFIG_NLS=yCONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_852=m CONFIG_NLS_CODEPAGE_1250=mCONFIG_NLS_ASCII=yCONFIG_NLS_ISO8859_1=mCONFIG_NLS_ISO8859_2=mCONFIG_NLS_UTF8=mCONFIG_DLM=m
CONFIG_TRACE_IRQFLAGS_SUPPORT=yCONFIG_PRINTK_TIME=yCONFIG_DEFAULT_MESSAGE_LOGLEVEL=4CONFIG_ENABLE_WARN_DEPRECATED=yCONFIG_ENABLE_MUST_CHECK=yCONFIG_FRAME_WARN=2048CONFIG_MAGIC_SYSRQ=yCONFIG_DEBUG_FS=yCONFIG_DEBUG_KERNEL=yCONFIG_LOCKUP_DETECTOR=yCONFIG_HARDLOCKUP_DETECTOR=yCONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0CONFIG_PANIC_ON_OOPS_VALUE=0CONFIG_DETECT_HUNG_TASK=yCONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0CONFIG_SCHED_DEBUG=yCONFIG_SCHEDSTATS=yCONFIG_TIMER_STATS=yCONFIG_SLUB_STATS=yCONFIG_DEBUG_KMEMLEAK=yCONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=yCONFIG_DEBUG_PREEMPT=yCONFIG_DEBUG_SPINLOCK=yCONFIG_DEBUG_MUTEXES=yCONFIG_DEBUG_LOCK_ALLOC=yCONFIG_PROVE_LOCKING=yCONFIG_PROVE_RCU=yCONFIG_LOCKDEP=yCONFIG_LOCK_STAT=yCONFIG_TRACE_IRQFLAGS=yCONFIG_DEBUG_ATOMIC_SLEEP=yCONFIG_STACKTRACE=yCONFIG_DEBUG_BUGVERBOSE=yCONFIG_DEBUG_INFO=yCONFIG_DEBUG_INFO_REDUCED=yCONFIG_DEBUG_MEMORY_INIT=yCONFIG_DEBUG_NOTIFIERS=yCONFIG_ARCH_WANT_FRAME_POINTERS=yCONFIG_FRAME_POINTER=yCONFIG_RCU_CPU_STALL_TIMEOUT=60CONFIG_RCU_CPU_STALL_VERBOSE=yCONFIG_RCU_CPU_STALL_INFO=yCONFIG_RCU_TRACE=yCONFIG_LKDTM=mCONFIG_LATENCYTOP=yCONFIG_USER_STACKTRACE_SUPPORT=yCONFIG_NOP_TRACER=yCONFIG_HAVE_FUNCTION_TRACER=yCONFIG_HAVE_FUNCTION_GRAPH_TRACER=yCONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=yCONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=yCONFIG_HAVE_DYNAMIC_FTRACE=yCONFIG_HAVE_FTRACE_MCOUNT_RECORD=yCONFIG_HAVE_SYSCALL_TRACEPOINTS=yCONFIG_HAVE_C_RECORDMCOUNT=yCONFIG_TRACER_MAX_TRACE=yCONFIG_RING_BUFFER=yCONFIG_EVENT_TRACING=yCONFIG_CONTEXT_SWITCH_TRACER=yCONFIG_RING_BUFFER_ALLOW_SWAP=yCONFIG_TRACING=yCONFIG_GENERIC_TRACER=yCONFIG_TRACING_SUPPORT=yCONFIG_FTRACE=yCONFIG_FUNCTION_TRACER=yCONFIG_FUNCTION_GRAPH_TRACER=yCONFIG_IRQSOFF_TRACER=yCONFIG_PREEMPT_TRACER=yCONFIG_SCHED_TRACER=yCONFIG_FTRACE_SYSCALLS=yCONFIG_BRANCH_PROFILE_NONE=yCONFIG_STACK_TRACER=yCONFIG_BLK_DEV_IO_TRACE=yCONFIG_KPROBE_EVENT=yCONFIG_UPROBE_EVENT=yCONFIG_PROBE_EVENTS=yCONFIG_DYNAMIC_FTRACE=yCONFIG_FUNCTION_PROFILER=yCONFIG_FTRACE_MCOUNT_RECORD=yCONFIG_MMIOTRACE=yCONFIG_MMIOTRACE_TEST=mCONFIG_DYNAMIC_DEBUG=yCONFIG_DMA_API_DEBUG=yCONFIG_HAVE_ARCH_KGDB=yCONFIG_HAVE_ARCH_KMEMCHECK=yCONFIG_X86_VERBOSE_BOOTUP=yCONFIG_EARLY_PRINTK=yCONFIG_DEBUG_RODATA=yCONFIG_HAVE_MMIOTRACE_SUPPORT=yCONFIG_IO_DELAY_TYPE_0X80=0CONFIG_IO_DELAY_TYPE_0XED=1CONFIG_IO_DELAY_TYPE_UDELAY=2CONFIG_IO_DELAY_TYPE_NONE=3CONFIG_IO_DELAY_0X80=yCONFIG_DEFAULT_IO_DELAY_TYPE=0CONFIG_DEBUG_BOOT_PARAMS=yCONFIG_OPTIMIZE_INLINING=y
CONFIG_KEYS=yCONFIG_ENCRYPTED_KEYS=yCONFIG_SECURITY=yCONFIG_SECURITY_NETWORK=yCONFIG_LSM_MMAP_MIN_ADDR=65535CONFIG_SECURITY_SELINUX=yCONFIG_SECURITY_SELINUX_BOOTPARAM=yCONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1CONFIG_SECURITY_SELINUX_DISABLE=yCONFIG_SECURITY_SELINUX_DEVELOP=yCONFIG_SECURITY_SELINUX_AVC_STATS=yCONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1CONFIG_INTEGRITY=yCONFIG_EVM=yCONFIG_DEFAULT_SECURITY_SELINUX=yCONFIG_DEFAULT_SECURITY="selinux"CONFIG_XOR_BLOCKS=mCONFIG_ASYNC_CORE=mCONFIG_ASYNC_MEMCPY=mCONFIG_ASYNC_XOR=mCONFIG_ASYNC_PQ=mCONFIG_ASYNC_RAID6_RECOV=mCONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=yCONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=yCONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=yCONFIG_CRYPTO_ALGAPI2=yCONFIG_CRYPTO_AEAD=mCONFIG_CRYPTO_AEAD2=yCONFIG_CRYPTO_BLKCIPHER=yCONFIG_CRYPTO_BLKCIPHER2=yCONFIG_CRYPTO_HASH=yCONFIG_CRYPTO_HASH2=yCONFIG_CRYPTO_RNG=yCONFIG_CRYPTO_RNG2=yCONFIG_CRYPTO_PCOMP2=yCONFIG_CRYPTO_MANAGER=yCONFIG_CRYPTO_MANAGER2=yCONFIG_CRYPTO_USER=mCONFIG_CRYPTO_MANAGER_DISABLE_TESTS=yCONFIG_CRYPTO_GF128MUL=mCONFIG_CRYPTO_PCRYPT=mCONFIG_CRYPTO_WORKQUEUE=yCONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_CBC=yCONFIG_CRYPTO_ECB=mCONFIG_CRYPTO_LRW=mCONFIG_CRYPTO_XTS=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_CRC32C=yCONFIG_CRYPTO_MD4=mCONFIG_CRYPTO_MD5=yCONFIG_CRYPTO_MICHAEL_MIC=mCONFIG_CRYPTO_SHA1=yCONFIG_CRYPTO_SHA1_SSSE3=yCONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_AES=yCONFIG_CRYPTO_AES_X86_64=mCONFIG_CRYPTO_AES_NI_INTEL=mCONFIG_CRYPTO_ARC4=mCONFIG_CRYPTO_BLOWFISH_COMMON=mCONFIG_CRYPTO_BLOWFISH_X86_64=mCONFIG_CRYPTO_DES=mCONFIG_CRYPTO_TWOFISH_COMMON=mCONFIG_CRYPTO_TWOFISH_X86_64=mCONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_HW=yCONFIG_HAVE_KVM=yCONFIG_HAVE_KVM_IRQCHIP=yCONFIG_HAVE_KVM_EVENTFD=yCONFIG_KVM_APIC_ARCHITECTURE=yCONFIG_KVM_MMIO=yCONFIG_KVM_ASYNC_PF=yCONFIG_HAVE_KVM_MSI=yCONFIG_VIRTUALIZATION=yCONFIG_KVM=mCONFIG_KVM_INTEL=mCONFIG_KVM_AMD=mCONFIG_KVM_MMU_AUDIT=yCONFIG_VHOST_NET=mCONFIG_BINARY_PRINTF=y
CONFIG_RAID6_PQ=mCONFIG_BITREVERSE=yCONFIG_GENERIC_STRNCPY_FROM_USER=yCONFIG_GENERIC_STRNLEN_USER=yCONFIG_GENERIC_FIND_FIRST_BIT=yCONFIG_GENERIC_PCI_IOMAP=yCONFIG_GENERIC_IOMAP=yCONFIG_GENERIC_IO=yCONFIG_CRC_CCITT=mCONFIG_CRC16=yCONFIG_CRC_T10DIF=yCONFIG_CRC_ITU_T=mCONFIG_CRC32=yCONFIG_CRC32_SLICEBY8=yCONFIG_LIBCRC32C=mCONFIG_ZLIB_INFLATE=yCONFIG_ZLIB_DEFLATE=mCONFIG_LZO_COMPRESS=yCONFIG_LZO_DECOMPRESS=yCONFIG_XZ_DEC=yCONFIG_XZ_DEC_X86=yCONFIG_XZ_DEC_POWERPC=yCONFIG_XZ_DEC_IA64=yCONFIG_XZ_DEC_ARM=yCONFIG_XZ_DEC_ARMTHUMB=yCONFIG_XZ_DEC_SPARC=yCONFIG_XZ_DEC_BCJ=yCONFIG_DECOMPRESS_GZIP=yCONFIG_DECOMPRESS_BZIP2=yCONFIG_DECOMPRESS_LZMA=yCONFIG_DECOMPRESS_XZ=yCONFIG_DECOMPRESS_LZO=yCONFIG_TEXTSEARCH=yCONFIG_TEXTSEARCH_KMP=mCONFIG_TEXTSEARCH_BM=mCONFIG_TEXTSEARCH_FSM=mCONFIG_HAS_IOMEM=yCONFIG_HAS_IOPORT=yCONFIG_HAS_DMA=yCONFIG_CHECK_SIGNATURE=yCONFIG_CPU_RMAP=yCONFIG_DQL=yCONFIG_NLATTR=yCONFIG_LRU_CACHE=mCONFIG_AVERAGE=yCONFIG_DDR=y
Hi
When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.
Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
My HW - t61, 4GB
Zdenek
BUG: unable to handle kernel NULL pointer dereference at 0000000000000178 IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] PGD 0 Oops: 0000 [#1] PREEMPT SMP CPU 1 Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight
Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO
RIP: 0010:[<ffffffffa031204d>] [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP: 0018:ffff880126b77c68 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607 RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40 RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400 R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task ffff8801344ca310) Stack: ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0 Call Trace: [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror] [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror] [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod] [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod] [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod] [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod] [<ffffffff81060065>] process_one_work+0x1a5/0x6b0 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod] [<ffffffff810608ee>] worker_thread+0x12e/0x2d0 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210 [<ffffffff81069cf2>] kthread+0xa2/0xb0 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0 [<ffffffff81533330>] ? gs_change+0xb/0xb Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 RIP [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP <ffff880126b77c68>
CR2: 0000000000000178
# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug fs entries on failure of file creation and made extfrag_debug_root dentry local git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105
CONFIG_64BIT=yCONFIG_X86_64=yCONFIG_X86=yCONFIG_INSTRUCTION_DECODER=yCONFIG_OUTPUT_FORMAT="elf64-x86-64"CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"CONFIG_LOCKDEP_SUPPORT=yCONFIG_STACKTRACE_SUPPORT=yCONFIG_HAVE_LATENCYTOP_SUPPORT=yCONFIG_MMU=yCONFIG_NEED_DMA_MAP_STATE=yCONFIG_NEED_SG_DMA_LENGTH=yCONFIG_GENERIC_ISA_DMA=yCONFIG_GENERIC_BUG=yCONFIG_GENERIC_BUG_RELATIVE_POINTERS=yCONFIG_GENERIC_HWEIGHT=yCONFIG_ARCH_MAY_HAVE_PC_FDC=yCONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_GENERIC_CALIBRATE_DELAY=yCONFIG_ARCH_HAS_CPU_RELAX=yCONFIG_ARCH_HAS_DEFAULT_IDLE=yCONFIG_ARCH_HAS_CACHE_LINE_SIZE=yCONFIG_ARCH_HAS_CPU_AUTOPROBE=yCONFIG_HAVE_SETUP_PER_CPU_AREA=yCONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=yCONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=yCONFIG_ARCH_HIBERNATION_POSSIBLE=yCONFIG_ARCH_SUSPEND_POSSIBLE=yCONFIG_ZONE_DMA32=yCONFIG_AUDIT_ARCH=yCONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=yCONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=yCONFIG_X86_64_SMP=yCONFIG_X86_HT=yCONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" CONFIG_ARCH_CPU_PROBE_RELEASE=yCONFIG_ARCH_SUPPORTS_UPROBES=yCONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"CONFIG_HAVE_IRQ_WORK=yCONFIG_IRQ_WORK=yCONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_EXPERIMENTAL=yCONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""CONFIG_LOCALVERSION=""CONFIG_LOCALVERSION_AUTO=yCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_BZIP2=yCONFIG_HAVE_KERNEL_LZMA=yCONFIG_HAVE_KERNEL_XZ=yCONFIG_HAVE_KERNEL_LZO=yCONFIG_KERNEL_LZMA=yCONFIG_DEFAULT_HOSTNAME="(none)"CONFIG_SWAP=yCONFIG_SYSVIPC=yCONFIG_SYSVIPC_SYSCTL=yCONFIG_POSIX_MQUEUE=yCONFIG_POSIX_MQUEUE_SYSCTL=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_TASKSTATS=yCONFIG_TASK_DELAY_ACCT=yCONFIG_TASK_XACCT=yCONFIG_TASK_IO_ACCOUNTING=yCONFIG_AUDIT=yCONFIG_AUDITSYSCALL=yCONFIG_AUDIT_WATCH=yCONFIG_AUDIT_TREE=yCONFIG_HAVE_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS=yCONFIG_GENERIC_IRQ_PROBE=yCONFIG_GENERIC_IRQ_SHOW=yCONFIG_GENERIC_PENDING_IRQ=yCONFIG_IRQ_FORCED_THREADING=yCONFIG_SPARSE_IRQ=yCONFIG_CLOCKSOURCE_WATCHDOG=yCONFIG_ARCH_CLOCKSOURCE_DATA=yCONFIG_GENERIC_TIME_VSYSCALL=yCONFIG_GENERIC_CLOCKEVENTS=yCONFIG_GENERIC_CLOCKEVENTS_BUILD=yCONFIG_GENERIC_CLOCKEVENTS_BROADCAST=yCONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=yCONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_TICK_ONESHOT=yCONFIG_NO_HZ=yCONFIG_HIGH_RES_TIMERS=y
CONFIG_TREE_PREEMPT_RCU=yCONFIG_PREEMPT_RCU=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16CONFIG_RCU_FAST_NO_HZ=yCONFIG_TREE_RCU_TRACE=yCONFIG_IKCONFIG=mCONFIG_IKCONFIG_PROC=yCONFIG_LOG_BUF_SHIFT=17CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=yCONFIG_CGROUPS=yCONFIG_CGROUP_FREEZER=yCONFIG_CGROUP_DEVICE=yCONFIG_CPUSETS=yCONFIG_PROC_PID_CPUSET=yCONFIG_CGROUP_CPUACCT=yCONFIG_RESOURCE_COUNTERS=yCONFIG_CGROUP_SCHED=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_CFS_BANDWIDTH=yCONFIG_RT_GROUP_SCHED=yCONFIG_BLK_CGROUP=yCONFIG_NAMESPACES=yCONFIG_UTS_NS=yCONFIG_IPC_NS=yCONFIG_PID_NS=yCONFIG_NET_NS=yCONFIG_SCHED_AUTOGROUP=yCONFIG_SYSFS_DEPRECATED=yCONFIG_RELAY=yCONFIG_BLK_DEV_INITRD=yCONFIG_INITRAMFS_SOURCE=""CONFIG_RD_GZIP=yCONFIG_RD_BZIP2=yCONFIG_RD_LZMA=yCONFIG_RD_XZ=yCONFIG_RD_LZO=yCONFIG_SYSCTL=yCONFIG_ANON_INODES=yCONFIG_UID16=yCONFIG_KALLSYMS=yCONFIG_KALLSYMS_ALL=yCONFIG_HOTPLUG=yCONFIG_PRINTK=yCONFIG_BUG=yCONFIG_ELF_CORE=yCONFIG_PCSPKR_PLATFORM=yCONFIG_HAVE_PCSPKR_PLATFORM=yCONFIG_BASE_FULL=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_SIGNALFD=yCONFIG_TIMERFD=yCONFIG_EVENTFD=yCONFIG_SHMEM=yCONFIG_AIO=yCONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_EVENTS=yCONFIG_VM_EVENT_COUNTERS=yCONFIG_PCI_QUIRKS=yCONFIG_SLUB_DEBUG=yCONFIG_SLUB=yCONFIG_PROFILING=yCONFIG_TRACEPOINTS=yCONFIG_OPROFILE=mCONFIG_HAVE_OPROFILE=yCONFIG_OPROFILE_NMI_TIMER=yCONFIG_KPROBES=yCONFIG_JUMP_LABEL=yCONFIG_UPROBES=yCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=yCONFIG_KRETPROBES=yCONFIG_USER_RETURN_NOTIFIER=yCONFIG_HAVE_IOREMAP_PROT=yCONFIG_HAVE_KPROBES=yCONFIG_HAVE_KRETPROBES=yCONFIG_HAVE_OPTPROBES=yCONFIG_HAVE_ARCH_TRACEHOOK=yCONFIG_HAVE_DMA_ATTRS=yCONFIG_USE_GENERIC_SMP_HELPERS=yCONFIG_GENERIC_SMP_IDLE_THREAD=yCONFIG_HAVE_REGS_AND_STACK_ACCESS_API=yCONFIG_HAVE_DMA_API_DEBUG=yCONFIG_HAVE_HW_BREAKPOINT=yCONFIG_HAVE_MIXED_BREAKPOINTS_REGS=yCONFIG_HAVE_USER_RETURN_NOTIFIER=yCONFIG_HAVE_PERF_EVENTS_NMI=yCONFIG_HAVE_ARCH_JUMP_LABEL=yCONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=yCONFIG_HAVE_ALIGNED_STRUCT_PAGE=yCONFIG_HAVE_CMPXCHG_LOCAL=yCONFIG_HAVE_CMPXCHG_DOUBLE=yCONFIG_ARCH_WANT_OLD_COMPAT_IPC=yCONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SLABINFO=yCONFIG_RT_MUTEXES=yCONFIG_BASE_SMALL=0CONFIG_MODULES=yCONFIG_MODULE_UNLOAD=yCONFIG_MODULE_FORCE_UNLOAD=yCONFIG_MODULE_SRCVERSION_ALL=yCONFIG_STOP_MACHINE=yCONFIG_BLOCK=yCONFIG_BLK_DEV_BSG=yCONFIG_BLK_DEV_INTEGRITY=yCONFIG_BLK_DEV_THROTTLING=y
CONFIG_PARTITION_ADVANCED=yCONFIG_AMIGA_PARTITION=yCONFIG_MSDOS_PARTITION=yCONFIG_LDM_PARTITION=yCONFIG_EFI_PARTITION=yCONFIG_BLOCK_COMPAT=y
CONFIG_IOSCHED_NOOP=yCONFIG_IOSCHED_DEADLINE=yCONFIG_IOSCHED_CFQ=mCONFIG_DEFAULT_DEADLINE=yCONFIG_DEFAULT_IOSCHED="deadline"CONFIG_PREEMPT_NOTIFIERS=yCONFIG_PADATA=yCONFIG_UNINLINE_SPIN_UNLOCK=yCONFIG_FREEZER=y
CONFIG_ZONE_DMA=yCONFIG_SMP=yCONFIG_X86_MPPARSE=yCONFIG_X86_SUPPORTS_MEMORY_FAILURE=yCONFIG_SCHED_OMIT_FRAME_POINTER=yCONFIG_NO_BOOTMEM=yCONFIG_MEMTEST=yCONFIG_MCORE2=yCONFIG_X86_INTERNODE_CACHE_SHIFT=6CONFIG_X86_CMPXCHG=yCONFIG_X86_L1_CACHE_SHIFT=6CONFIG_X86_XADD=yCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INTEL_USERCOPY=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_X86_P6_NOP=yCONFIG_X86_TSC=yCONFIG_X86_CMPXCHG64=yCONFIG_X86_CMOV=yCONFIG_X86_MINIMUM_CPU_FAMILY=64CONFIG_X86_DEBUGCTLMSR=yCONFIG_CPU_SUP_INTEL=yCONFIG_CPU_SUP_AMD=yCONFIG_CPU_SUP_CENTAUR=yCONFIG_HPET_TIMER=yCONFIG_HPET_EMULATE_RTC=yCONFIG_DMI=yCONFIG_GART_IOMMU=yCONFIG_SWIOTLB=yCONFIG_IOMMU_HELPER=yCONFIG_NR_CPUS=2CONFIG_PREEMPT=yCONFIG_PREEMPT_COUNT=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=yCONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=yCONFIG_X86_MCE=yCONFIG_X86_MCE_INTEL=yCONFIG_X86_MCE_AMD=yCONFIG_X86_MCE_THRESHOLD=yCONFIG_X86_THERMAL_VECTOR=yCONFIG_MICROCODE=mCONFIG_MICROCODE_INTEL=yCONFIG_MICROCODE_AMD=yCONFIG_MICROCODE_OLD_INTERFACE=yCONFIG_X86_MSR=mCONFIG_X86_CPUID=mCONFIG_ARCH_PHYS_ADDR_T_64BIT=yCONFIG_ARCH_DMA_ADDR_T_64BIT=yCONFIG_DIRECT_GBPAGES=yCONFIG_ARCH_SPARSEMEM_ENABLE=yCONFIG_ARCH_SPARSEMEM_DEFAULT=yCONFIG_ARCH_SELECT_MEMORY_MODEL=yCONFIG_ARCH_PROC_KCORE_TEXT=yCONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000CONFIG_SELECT_MEMORY_MODEL=yCONFIG_SPARSEMEM_MANUAL=yCONFIG_SPARSEMEM=yCONFIG_HAVE_MEMORY_PRESENT=yCONFIG_SPARSEMEM_EXTREME=yCONFIG_SPARSEMEM_VMEMMAP_ENABLE=yCONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=yCONFIG_SPARSEMEM_VMEMMAP=yCONFIG_HAVE_MEMBLOCK=yCONFIG_HAVE_MEMBLOCK_NODE_MAP=yCONFIG_ARCH_DISCARD_MEMBLOCK=yCONFIG_PAGEFLAGS_EXTENDED=yCONFIG_SPLIT_PTLOCK_CPUS=999999CONFIG_COMPACTION=yCONFIG_MIGRATION=yCONFIG_PHYS_ADDR_T_64BIT=yCONFIG_ZONE_DMA_FLAG=1CONFIG_BOUNCE=yCONFIG_VIRT_TO_BUS=yCONFIG_MMU_NOTIFIER=yCONFIG_KSM=yCONFIG_DEFAULT_MMAP_MIN_ADDR=4096CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=yCONFIG_CROSS_MEMORY_ATTACH=yCONFIG_CLEANCACHE=yCONFIG_FRONTSWAP=yCONFIG_X86_CHECK_BIOS_CORRUPTION=yCONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=yCONFIG_X86_RESERVE_LOW=64CONFIG_MTRR=yCONFIG_MTRR_SANITIZER=yCONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1CONFIG_X86_PAT=yCONFIG_ARCH_USES_PG_UNCACHED=yCONFIG_ARCH_RANDOM=yCONFIG_EFI=yCONFIG_HZ_300=yCONFIG_HZ=300CONFIG_SCHED_HRTICK=yCONFIG_KEXEC=yCONFIG_KEXEC_JUMP=yCONFIG_PHYSICAL_START=0x1000000CONFIG_PHYSICAL_ALIGN=0x1000000CONFIG_HOTPLUG_CPU=yCONFIG_CMDLINE_BOOL=yCONFIG_CMDLINE=""CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_HIBERNATION_HEADER=yCONFIG_SUSPEND=yCONFIG_SUSPEND_FREEZER=yCONFIG_HIBERNATE_CALLBACKS=yCONFIG_HIBERNATION=yCONFIG_PM_STD_PARTITION=""CONFIG_PM_SLEEP=yCONFIG_PM_SLEEP_SMP=yCONFIG_PM_WAKELOCKS=yCONFIG_PM_WAKELOCKS_LIMIT=100CONFIG_PM_WAKELOCKS_GC=yCONFIG_PM_RUNTIME=yCONFIG_PM=yCONFIG_PM_DEBUG=yCONFIG_PM_ADVANCED_DEBUG=yCONFIG_CAN_PM_TRACE=yCONFIG_PM_TRACE=yCONFIG_PM_TRACE_RTC=yCONFIG_ACPI=yCONFIG_ACPI_SLEEP=yCONFIG_ACPI_PROCFS=yCONFIG_ACPI_PROCFS_POWER=yCONFIG_ACPI_AC=yCONFIG_ACPI_BATTERY=yCONFIG_ACPI_BUTTON=yCONFIG_ACPI_VIDEO=mCONFIG_ACPI_FAN=yCONFIG_ACPI_DOCK=yCONFIG_ACPI_PROCESSOR=yCONFIG_ACPI_HOTPLUG_CPU=yCONFIG_ACPI_PROCESSOR_AGGREGATOR=mCONFIG_ACPI_THERMAL=yCONFIG_ACPI_BLACKLIST_YEAR=0CONFIG_ACPI_DEBUG=yCONFIG_ACPI_PCI_SLOT=yCONFIG_X86_PM_TIMER=yCONFIG_ACPI_CONTAINER=yCONFIG_ACPI_CUSTOM_METHOD=m
CONFIG_CPU_FREQ=yCONFIG_CPU_FREQ_TABLE=yCONFIG_CPU_FREQ_STAT=yCONFIG_CPU_FREQ_STAT_DETAILS=yCONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=yCONFIG_CPU_FREQ_GOV_PERFORMANCE=yCONFIG_CPU_FREQ_GOV_POWERSAVE=yCONFIG_CPU_FREQ_GOV_USERSPACE=yCONFIG_CPU_FREQ_GOV_ONDEMAND=yCONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_X86_PCC_CPUFREQ=mCONFIG_X86_ACPI_CPUFREQ=yCONFIG_X86_P4_CLOCKMOD=m
CONFIG_X86_SPEEDSTEP_LIB=mCONFIG_CPU_IDLE=yCONFIG_CPU_IDLE_GOV_LADDER=yCONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_PCI=yCONFIG_PCI_DIRECT=yCONFIG_PCI_MMCONFIG=yCONFIG_PCI_DOMAINS=yCONFIG_PCIEPORTBUS=yCONFIG_PCIEAER=yCONFIG_PCIEASPM=yCONFIG_PCIEASPM_DEFAULT=yCONFIG_PCIE_PME=yCONFIG_ARCH_SUPPORTS_MSI=yCONFIG_PCI_MSI=yCONFIG_PCI_STUB=yCONFIG_HT_IRQ=yCONFIG_PCI_ATS=yCONFIG_PCI_IOV=yCONFIG_PCI_PRI=yCONFIG_PCI_PASID=yCONFIG_PCI_IOAPIC=yCONFIG_PCI_LABEL=yCONFIG_ISA_DMA_API=yCONFIG_AMD_NB=yCONFIG_PCCARD=yCONFIG_PCMCIA=mCONFIG_PCMCIA_LOAD_CIS=yCONFIG_CARDBUS=y
CONFIG_YENTA=mCONFIG_YENTA_O2=yCONFIG_YENTA_RICOH=yCONFIG_YENTA_TI=yCONFIG_YENTA_ENE_TUNE=yCONFIG_YENTA_TOSHIBA=yCONFIG_PCCARD_NONSTATIC=yCONFIG_HOTPLUG_PCI=y
CONFIG_BINFMT_ELF=yCONFIG_COMPAT_BINFMT_ELF=yCONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=yCONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=yCONFIG_BINFMT_MISC=mCONFIG_IA32_EMULATION=yCONFIG_IA32_AOUT=mCONFIG_X86_X32=yCONFIG_COMPAT=yCONFIG_COMPAT_FOR_U64_ALIGNMENT=yCONFIG_SYSVIPC_COMPAT=yCONFIG_KEYS_COMPAT=yCONFIG_HAVE_TEXT_POKE_SMP=yCONFIG_X86_DEV_DMA_OPS=yCONFIG_NET=yCONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_PACKET=yCONFIG_UNIX=yCONFIG_UNIX_DIAG=mCONFIG_XFRM=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_PNP=yCONFIG_IP_PNP_DHCP=yCONFIG_IP_PNP_BOOTP=yCONFIG_IP_MROUTE=yCONFIG_SYN_COOKIES=yCONFIG_INET_TUNNEL=mCONFIG_TCP_CONG_ADVANCED=yCONFIG_TCP_CONG_BIC=mCONFIG_TCP_CONG_CUBIC=yCONFIG_TCP_CONG_WESTWOOD=mCONFIG_TCP_CONG_HTCP=mCONFIG_DEFAULT_CUBIC=yCONFIG_DEFAULT_TCP_CONG="cubic"CONFIG_IPV6=mCONFIG_INET6_XFRM_MODE_TRANSPORT=mCONFIG_INET6_XFRM_MODE_TUNNEL=mCONFIG_INET6_XFRM_MODE_BEET=mCONFIG_IPV6_SIT=mCONFIG_IPV6_NDISC_NODETYPE=yCONFIG_NETWORK_SECMARK=yCONFIG_NETFILTER=yCONFIG_NETFILTER_ADVANCED=yCONFIG_BRIDGE_NETFILTER=y
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_NETLINK_QUEUE=mCONFIG_NETFILTER_NETLINK_LOG=mCONFIG_NF_CONNTRACK=mCONFIG_NF_CONNTRACK_MARK=yCONFIG_NF_CONNTRACK_SECMARK=yCONFIG_NF_CONNTRACK_PROCFS=yCONFIG_NF_CONNTRACK_EVENTS=yCONFIG_NF_CONNTRACK_FTP=mCONFIG_NF_CT_NETLINK=mCONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_MARK=mCONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=mCONFIG_NETFILTER_XT_TARGET_CONNMARK=mCONFIG_NETFILTER_XT_TARGET_CONNSECMARK=mCONFIG_NETFILTER_XT_TARGET_DSCP=mCONFIG_NETFILTER_XT_TARGET_HL=mCONFIG_NETFILTER_XT_TARGET_LED=mCONFIG_NETFILTER_XT_TARGET_MARK=mCONFIG_NETFILTER_XT_TARGET_NFLOG=mCONFIG_NETFILTER_XT_TARGET_NFQUEUE=mCONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=mCONFIG_NETFILTER_XT_MATCH_CONNLIMIT=mCONFIG_NETFILTER_XT_MATCH_CONNMARK=mCONFIG_NETFILTER_XT_MATCH_CONNTRACK=mCONFIG_NETFILTER_XT_MATCH_DSCP=mCONFIG_NETFILTER_XT_MATCH_ECN=mCONFIG_NETFILTER_XT_MATCH_HL=mCONFIG_NETFILTER_XT_MATCH_IPRANGE=mCONFIG_NETFILTER_XT_MATCH_MARK=mCONFIG_NETFILTER_XT_MATCH_MULTIPORT=mCONFIG_NETFILTER_XT_MATCH_OSF=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NETFILTER_XT_MATCH_POLICY=mCONFIG_NETFILTER_XT_MATCH_PHYSDEV=mCONFIG_NETFILTER_XT_MATCH_PKTTYPE=mCONFIG_NETFILTER_XT_MATCH_QUOTA=mCONFIG_NETFILTER_XT_MATCH_SCTP=mCONFIG_NETFILTER_XT_MATCH_STATE=mCONFIG_NETFILTER_XT_MATCH_STATISTIC=mCONFIG_NETFILTER_XT_MATCH_STRING=mCONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NF_DEFRAG_IPV4=mCONFIG_NF_CONNTRACK_IPV4=mCONFIG_NF_CONNTRACK_PROC_COMPAT=yCONFIG_IP_NF_IPTABLES=mCONFIG_IP_NF_MATCH_AH=mCONFIG_IP_NF_MATCH_ECN=mCONFIG_IP_NF_MATCH_TTL=mCONFIG_IP_NF_FILTER=mCONFIG_IP_NF_TARGET_REJECT=mCONFIG_IP_NF_TARGET_ULOG=mCONFIG_NF_NAT=mCONFIG_NF_NAT_NEEDED=yCONFIG_IP_NF_TARGET_MASQUERADE=mCONFIG_IP_NF_TARGET_REDIRECT=mCONFIG_NF_NAT_FTP=mCONFIG_IP_NF_MANGLE=mCONFIG_IP_NF_TARGET_ECN=mCONFIG_IP_NF_TARGET_TTL=mCONFIG_IP_NF_ARPTABLES=mCONFIG_IP_NF_ARPFILTER=mCONFIG_IP_NF_ARP_MANGLE=m
CONFIG_NF_DEFRAG_IPV6=mCONFIG_NF_CONNTRACK_IPV6=mCONFIG_IP6_NF_IPTABLES=mCONFIG_BRIDGE_NF_EBTABLES=mCONFIG_BRIDGE_EBT_T_NAT=mCONFIG_IP_SCTP=mCONFIG_SCTP_HMAC_NONE=yCONFIG_STP=mCONFIG_BRIDGE=mCONFIG_BRIDGE_IGMP_SNOOPING=yCONFIG_LLC=mCONFIG_DNS_RESOLVER=yCONFIG_RPS=yCONFIG_RFS_ACCEL=yCONFIG_XPS=yCONFIG_NETPRIO_CGROUP=mCONFIG_BQL=y
CONFIG_NET_PKTGEN=mCONFIG_IRDA=m
CONFIG_IRTTY_SIR=m
CONFIG_BT=mCONFIG_BT_RFCOMM=mCONFIG_BT_RFCOMM_TTY=yCONFIG_BT_BNEP=mCONFIG_BT_HIDP=m
CONFIG_BT_HCIBTUSB=mCONFIG_WIRELESS=yCONFIG_WEXT_CORE=yCONFIG_WEXT_PROC=yCONFIG_CFG80211=mCONFIG_CFG80211_DEVELOPER_WARNINGS=yCONFIG_CFG80211_DEFAULT_PS=yCONFIG_CFG80211_DEBUGFS=yCONFIG_CFG80211_WEXT=yCONFIG_WIRELESS_EXT_SYSFS=yCONFIG_LIB80211=mCONFIG_MAC80211=mCONFIG_MAC80211_HAS_RC=yCONFIG_MAC80211_RC_MINSTREL=yCONFIG_MAC80211_RC_MINSTREL_HT=yCONFIG_MAC80211_RC_DEFAULT_MINSTREL=yCONFIG_MAC80211_RC_DEFAULT="minstrel_ht"CONFIG_MAC80211_LEDS=yCONFIG_MAC80211_DEBUGFS=yCONFIG_RFKILL=yCONFIG_RFKILL_LEDS=yCONFIG_RFKILL_INPUT=yCONFIG_RFKILL_REGULATOR=mCONFIG_HAVE_BPF_JIT=y
CONFIG_UEVENT_HELPER_PATH=""CONFIG_DEVTMPFS=yCONFIG_DEVTMPFS_MOUNT=yCONFIG_STANDALONE=yCONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_FW_LOADER=yCONFIG_FIRMWARE_IN_KERNEL=yCONFIG_EXTRA_FIRMWARE=""CONFIG_DMA_SHARED_BUFFER=yCONFIG_CONNECTOR=mCONFIG_MTD=mCONFIG_MTD_REDBOOT_PARTS=mCONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
CONFIG_MTD_CHAR=mCONFIG_MTD_BLKDEVS=mCONFIG_MTD_BLOCK=mCONFIG_SM_FTL=m
CONFIG_MTD_MAP_BANK_WIDTH_1=yCONFIG_MTD_MAP_BANK_WIDTH_2=yCONFIG_MTD_MAP_BANK_WIDTH_4=yCONFIG_MTD_CFI_I1=yCONFIG_MTD_CFI_I2=y
CONFIG_MTD_MTDRAM=mCONFIG_MTDRAM_TOTAL_SIZE=4096CONFIG_MTDRAM_ERASE_SIZE=128CONFIG_MTD_BLOCK2MTD=m
CONFIG_MTD_NAND_ECC=mCONFIG_MTD_NAND_ECC_SMC=y
CONFIG_PNP=y
CONFIG_PNPACPI=yCONFIG_BLK_DEV=yCONFIG_BLK_DEV_FD=mCONFIG_BLK_DEV_LOOP=mCONFIG_BLK_DEV_LOOP_MIN_COUNT=8CONFIG_BLK_DEV_CRYPTOLOOP=mCONFIG_BLK_DEV_DRBD=mCONFIG_BLK_DEV_NBD=mCONFIG_BLK_DEV_RAM=yCONFIG_BLK_DEV_RAM_COUNT=16CONFIG_BLK_DEV_RAM_SIZE=8192CONFIG_CDROM_PKTCDVD=mCONFIG_CDROM_PKTCDVD_BUFFERS=8CONFIG_VIRTIO_BLK=y
CONFIG_INTEL_MEI=mCONFIG_HAVE_IDE=y
CONFIG_SCSI_MOD=yCONFIG_RAID_ATTRS=mCONFIG_SCSI=yCONFIG_SCSI_DMA=y
CONFIG_BLK_DEV_SD=yCONFIG_BLK_DEV_SR=mCONFIG_BLK_DEV_SR_VENDOR=yCONFIG_CHR_DEV_SG=yCONFIG_CHR_DEV_SCH=mCONFIG_SCSI_MULTI_LUN=yCONFIG_SCSI_CONSTANTS=yCONFIG_SCSI_LOGGING=yCONFIG_SCSI_SCAN_ASYNC=yCONFIG_SCSI_WAIT_SCAN=m
CONFIG_SCSI_SPI_ATTRS=yCONFIG_SCSI_LOWLEVEL=yCONFIG_SCSI_DEBUG=mCONFIG_SCSI_VIRTIO=mCONFIG_ATA=yCONFIG_ATA_VERBOSE_ERROR=yCONFIG_ATA_ACPI=yCONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=yCONFIG_ATA_SFF=y
CONFIG_ATA_BMDMA=y
CONFIG_ATA_PIIX=y
CONFIG_PATA_MPIIX=y
CONFIG_MD=yCONFIG_BLK_DEV_MD=mCONFIG_MD_LINEAR=mCONFIG_MD_RAID0=mCONFIG_MD_RAID1=mCONFIG_MD_RAID10=mCONFIG_MD_RAID456=mCONFIG_MULTICORE_RAID456=yCONFIG_MD_MULTIPATH=mCONFIG_MD_FAULTY=mCONFIG_BLK_DEV_DM=mCONFIG_DM_DEBUG=yCONFIG_DM_BUFIO=mCONFIG_DM_PERSISTENT_DATA=mCONFIG_DM_CRYPT=mCONFIG_DM_SNAPSHOT=mCONFIG_DM_THIN_PROVISIONING=mCONFIG_DM_MIRROR=mCONFIG_DM_RAID=mCONFIG_DM_LOG_USERSPACE=mCONFIG_DM_ZERO=mCONFIG_DM_MULTIPATH=mCONFIG_DM_DELAY=mCONFIG_DM_UEVENT=yCONFIG_DM_FLAKEY=mCONFIG_DM_VERITY=m
CONFIG_NETDEVICES=yCONFIG_NET_CORE=yCONFIG_DUMMY=mCONFIG_MII=mCONFIG_NETCONSOLE=mCONFIG_NETPOLL=yCONFIG_NETPOLL_TRAP=yCONFIG_NET_POLL_CONTROLLER=yCONFIG_TUN=mCONFIG_VETH=mCONFIG_VIRTIO_NET=m
CONFIG_ETHERNET=yCONFIG_NET_VENDOR_INTEL=yCONFIG_E1000E=mCONFIG_IXGBEVF=mCONFIG_NET_VENDOR_REALTEK=yCONFIG_8139TOO=mCONFIG_8139TOO_PIO=yCONFIG_PPP=mCONFIG_PPP_BSDCOMP=mCONFIG_PPP_DEFLATE=mCONFIG_PPP_FILTER=yCONFIG_PPP_MPPE=mCONFIG_PPP_MULTILINK=yCONFIG_PPP_ASYNC=mCONFIG_SLHC=m
CONFIG_WLAN=yCONFIG_IWLWIFI=m
CONFIG_IWLEGACY=mCONFIG_IWL3945=m
CONFIG_IWLEGACY_DEBUGFS=y
CONFIG_INPUT=yCONFIG_INPUT_POLLDEV=yCONFIG_INPUT_SPARSEKMAP=m
CONFIG_INPUT_MOUSEDEV=yCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768CONFIG_INPUT_EVDEV=m
CONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ATKBD=yCONFIG_INPUT_MOUSE=yCONFIG_MOUSE_PS2=mCONFIG_MOUSE_PS2_ALPS=yCONFIG_MOUSE_PS2_LOGIPS2PP=yCONFIG_MOUSE_PS2_SYNAPTICS=yCONFIG_MOUSE_PS2_LIFEBOOK=yCONFIG_MOUSE_PS2_TRACKPOINT=yCONFIG_MOUSE_SERIAL=mCONFIG_INPUT_MISC=yCONFIG_INPUT_UINPUT=m
CONFIG_SERIO=yCONFIG_SERIO_I8042=yCONFIG_SERIO_SERPORT=mCONFIG_SERIO_PCIPS2=mCONFIG_SERIO_LIBPS2=yCONFIG_SERIO_RAW=m
CONFIG_VT=yCONFIG_CONSOLE_TRANSLATIONS=yCONFIG_VT_CONSOLE=yCONFIG_VT_CONSOLE_SLEEP=yCONFIG_HW_CONSOLE=yCONFIG_UNIX98_PTYS=yCONFIG_DEVKMEM=y
CONFIG_SERIAL_8250=yCONFIG_SERIAL_8250_CONSOLE=yCONFIG_FIX_EARLYCON_MEM=yCONFIG_SERIAL_8250_PCI=yCONFIG_SERIAL_8250_PNP=yCONFIG_SERIAL_8250_NR_UARTS=16CONFIG_SERIAL_8250_RUNTIME_UARTS=4CONFIG_SERIAL_8250_EXTENDED=yCONFIG_SERIAL_8250_SHARE_IRQ=yCONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_CORE=yCONFIG_SERIAL_CORE_CONSOLE=yCONFIG_HVC_DRIVER=yCONFIG_VIRTIO_CONSOLE=mCONFIG_HW_RANDOM=mCONFIG_HW_RANDOM_INTEL=mCONFIG_HW_RANDOM_VIRTIO=mCONFIG_NVRAM=m
CONFIG_HPET=yCONFIG_HPET_MMAP=yCONFIG_HANGCHECK_TIMER=mCONFIG_DEVPORT=yCONFIG_I2C=mCONFIG_I2C_BOARDINFO=yCONFIG_I2C_COMPAT=yCONFIG_I2C_CHARDEV=mCONFIG_I2C_HELPER_AUTO=yCONFIG_I2C_ALGOBIT=m
CONFIG_I2C_I801=mCONFIG_I2C_ISCH=mCONFIG_I2C_PIIX4=m
CONFIG_I2C_TINY_USB=m
CONFIG_I2C_DEBUG_ALGO=yCONFIG_HSI=mCONFIG_HSI_BOARDINFO=y
CONFIG_HSI_CHAR=m
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=yCONFIG_POWER_SUPPLY=yCONFIG_HWMON=y
CONFIG_SENSORS_ACPI_POWER=mCONFIG_THERMAL=yCONFIG_THERMAL_HWMON=yCONFIG_WATCHDOG=yCONFIG_WATCHDOG_CORE=y
CONFIG_SOFT_WATCHDOG=mCONFIG_ITCO_WDT=mCONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_SSB_POSSIBLE=y
CONFIG_BCMA_POSSIBLE=y
CONFIG_MFD_CORE=mCONFIG_LPC_SCH=mCONFIG_LPC_ICH=mCONFIG_REGULATOR=yCONFIG_REGULATOR_DEBUG=yCONFIG_REGULATOR_DUMMY=yCONFIG_REGULATOR_VIRTUAL_CONSUMER=mCONFIG_MEDIA_SUPPORT=m
CONFIG_VIDEO_DEV=mCONFIG_VIDEO_V4L2_COMMON=mCONFIG_DVB_CORE=mCONFIG_VIDEO_MEDIA=m
CONFIG_RC_CORE=mCONFIG_LIRC=mCONFIG_RC_MAP=mCONFIG_IR_SANYO_DECODER=mCONFIG_RC_LOOPBACK=mCONFIG_MEDIA_ATTACH=yCONFIG_MEDIA_TUNER=mCONFIG_MEDIA_TUNER_CUSTOMISE=y
CONFIG_MEDIA_TUNER_SIMPLE=mCONFIG_MEDIA_TUNER_TDA9887=mCONFIG_MEDIA_TUNER_MT2060=mCONFIG_MEDIA_TUNER_MT2266=mCONFIG_MEDIA_TUNER_TDA18218=mCONFIG_MEDIA_TUNER_TDA18212=mCONFIG_VIDEO_V4L2=mCONFIG_VIDEOBUF_GEN=mCONFIG_VIDEOBUF_DMA_SG=mCONFIG_VIDEO_TVEEPROM=mCONFIG_VIDEO_TUNER=mCONFIG_V4L2_MEM2MEM_DEV=mCONFIG_VIDEOBUF2_CORE=mCONFIG_VIDEOBUF2_MEMOPS=mCONFIG_VIDEOBUF2_VMALLOC=mCONFIG_VIDEO_CAPTURE_DRIVERS=yCONFIG_VIDEO_ADV_DEBUG=yCONFIG_VIDEO_HELPER_CHIPS_AUTO=yCONFIG_VIDEO_IR_I2C=m
CONFIG_VIDEO_MSP3400=mCONFIG_VIDEO_CS53L32A=mCONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_SAA6588=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_CX25840=m
CONFIG_VIDEO_CX2341X=m
CONFIG_VIDEO_VIVI=mCONFIG_V4L_USB_DRIVERS=yCONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=yCONFIG_VIDEO_PVRUSB2=mCONFIG_VIDEO_PVRUSB2_SYSFS=yCONFIG_VIDEO_PVRUSB2_DVB=yCONFIG_V4L_PCI_DRIVERS=yCONFIG_VIDEO_SAA7134=mCONFIG_VIDEO_SAA7134_ALSA=mCONFIG_VIDEO_SAA7134_RC=yCONFIG_V4L_MEM2MEM_DRIVERS=yCONFIG_VIDEO_MEM2MEM_TESTDEV=mCONFIG_DVB_MAX_ADAPTERS=8CONFIG_DVB_DYNAMIC_MINORS=yCONFIG_DVB_CAPTURE_DRIVERS=y
CONFIG_DVB_USB=mCONFIG_DVB_USB_DIB0700=mCONFIG_DVB_USB_CXUSB=mCONFIG_DVB_USB_NOVA_T_USB2=mCONFIG_DVB_USB_AF9015=mCONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_FE_CUSTOMISE=y
CONFIG_DVB_TDA8261=mCONFIG_DVB_TDA10071=m
CONFIG_DVB_DIB3000MC=mCONFIG_DVB_DIB7000M=mCONFIG_DVB_DIB7000P=mCONFIG_DVB_AF9013=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_PLL=mCONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_DUMMY_FE=m
CONFIG_AGP=yCONFIG_AGP_AMD64=yCONFIG_AGP_INTEL=yCONFIG_VGA_ARB=yCONFIG_VGA_ARB_MAX_GPUS=16CONFIG_DRM=mCONFIG_DRM_KMS_HELPER=mCONFIG_DRM_TTM=m
CONFIG_DRM_I915=mCONFIG_DRM_MGA=mCONFIG_DRM_CIRRUS_QEMU=mCONFIG_VIDEO_OUTPUT_CONTROL=mCONFIG_FB=yCONFIG_FIRMWARE_EDID=yCONFIG_FB_BOOT_VESA_SUPPORT=yCONFIG_FB_CFB_FILLRECT=yCONFIG_FB_CFB_COPYAREA=yCONFIG_FB_CFB_IMAGEBLIT=yCONFIG_FB_SYS_FILLRECT=mCONFIG_FB_SYS_COPYAREA=mCONFIG_FB_SYS_IMAGEBLIT=mCONFIG_FB_MODE_HELPERS=yCONFIG_FB_TILEBLITTING=y
CONFIG_FB_VESA=yCONFIG_FB_EFI=yCONFIG_BACKLIGHT_LCD_SUPPORT=yCONFIG_LCD_CLASS_DEVICE=mCONFIG_LCD_PLATFORM=mCONFIG_BACKLIGHT_CLASS_DEVICE=mCONFIG_BACKLIGHT_GENERIC=m
CONFIG_VGA_CONSOLE=yCONFIG_VGACON_SOFT_SCROLLBACK=yCONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64CONFIG_DUMMY_CONSOLE=yCONFIG_FRAMEBUFFER_CONSOLE=yCONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=yCONFIG_LOGO=yCONFIG_LOGO_LINUX_CLUT224=yCONFIG_SOUND=mCONFIG_SOUND_OSS_CORE=yCONFIG_SOUND_OSS_CORE_PRECLAIM=yCONFIG_SND=mCONFIG_SND_TIMER=mCONFIG_SND_PCM=mCONFIG_SND_HWDEP=mCONFIG_SND_RAWMIDI=mCONFIG_SND_JACK=yCONFIG_SND_SEQUENCER=mCONFIG_SND_OSSEMUL=yCONFIG_SND_MIXER_OSS=mCONFIG_SND_PCM_OSS=mCONFIG_SND_PCM_OSS_PLUGINS=yCONFIG_SND_SEQUENCER_OSS=yCONFIG_SND_HRTIMER=mCONFIG_SND_SEQ_HRTIMER_DEFAULT=yCONFIG_SND_DYNAMIC_MINORS=yCONFIG_SND_VERBOSE_PROCFS=yCONFIG_SND_VMASTER=yCONFIG_SND_KCTL_JACK=yCONFIG_SND_DMA_SGBUF=yCONFIG_SND_RAWMIDI_SEQ=mCONFIG_SND_EMU10K1_SEQ=mCONFIG_SND_MPU401_UART=mCONFIG_SND_AC97_CODEC=mCONFIG_SND_DRIVERS=yCONFIG_SND_DUMMY=mCONFIG_SND_SERIAL_U16550=mCONFIG_SND_MPU401=mCONFIG_SND_AC97_POWER_SAVE=yCONFIG_SND_AC97_POWER_SAVE_DEFAULT=0CONFIG_SND_PCI=yCONFIG_SND_EMU10K1=mCONFIG_SND_HDA_INTEL=mCONFIG_SND_HDA_PREALLOC_SIZE=64CONFIG_SND_HDA_INPUT_JACK=yCONFIG_SND_HDA_CODEC_REALTEK=yCONFIG_SND_HDA_CODEC_ANALOG=yCONFIG_SND_HDA_CODEC_SIGMATEL=yCONFIG_SND_HDA_CODEC_VIA=yCONFIG_SND_HDA_CODEC_HDMI=yCONFIG_SND_HDA_CODEC_CIRRUS=yCONFIG_SND_HDA_CODEC_CONEXANT=yCONFIG_SND_HDA_CODEC_CMEDIA=yCONFIG_SND_HDA_CODEC_SI3054=yCONFIG_SND_HDA_GENERIC=yCONFIG_SND_HDA_POWER_SAVE=yCONFIG_SND_HDA_POWER_SAVE_DEFAULT=500CONFIG_SND_INTEL8X0=mCONFIG_AC97_BUS=m
CONFIG_HID=mCONFIG_HID_GENERIC=m
CONFIG_HID_A4TECH=mCONFIG_HID_APPLE=mCONFIG_HID_BELKIN=mCONFIG_HID_CHERRY=mCONFIG_HID_CHICONY=mCONFIG_HID_CYPRESS=mCONFIG_HID_EZKEY=mCONFIG_HID_KYE=mCONFIG_HID_KENSINGTON=mCONFIG_HID_LOGITECH=mCONFIG_HID_MICROSOFT=mCONFIG_HID_MONTEREY=m
CONFIG_USB_HID=mCONFIG_USB_HIDDEV=yCONFIG_USB_ARCH_HAS_OHCI=yCONFIG_USB_ARCH_HAS_EHCI=yCONFIG_USB_ARCH_HAS_XHCI=yCONFIG_USB_SUPPORT=yCONFIG_USB_COMMON=mCONFIG_USB_ARCH_HAS_HCD=yCONFIG_USB=mCONFIG_USB_ANNOUNCE_NEW_DEVICES=y
CONFIG_USB_SUSPEND=yCONFIG_USB_MON=m
CONFIG_USB_XHCI_HCD=mCONFIG_USB_EHCI_HCD=mCONFIG_USB_EHCI_ROOT_HUB_TT=yCONFIG_USB_EHCI_TT_NEWSCHED=yCONFIG_USB_OHCI_HCD=mCONFIG_USB_OHCI_LITTLE_ENDIAN=yCONFIG_USB_UHCI_HCD=m
CONFIG_USB_ACM=mCONFIG_USB_PRINTER=mCONFIG_USB_WDM=mCONFIG_USB_TMC=m
CONFIG_USB_STORAGE=mCONFIG_USB_UAS=m
CONFIG_USB_SERIAL=mCONFIG_USB_SERIAL_GENERIC=yCONFIG_USB_SERIAL_IR=mCONFIG_USB_SERIAL_WWAN=mCONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_TEST=m
CONFIG_MMC=m
CONFIG_MMC_BLOCK=mCONFIG_MMC_BLOCK_MINORS=8CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_MMC_SDHCI=mCONFIG_MMC_SDHCI_PCI=mCONFIG_MMC_SDHCI_PLTFM=mCONFIG_MMC_SDRICOH_CS=mCONFIG_NEW_LEDS=yCONFIG_LEDS_CLASS=y
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=mCONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_EDAC=y
CONFIG_EDAC_DEBUG=yCONFIG_EDAC_DECODE_MCE=yCONFIG_EDAC_MM_EDAC=mCONFIG_EDAC_E752X=mCONFIG_RTC_LIB=yCONFIG_RTC_CLASS=yCONFIG_RTC_HCTOSYS=yCONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_INTF_SYSFS=yCONFIG_RTC_INTF_PROC=yCONFIG_RTC_INTF_DEV=y
CONFIG_RTC_DRV_CMOS=y
CONFIG_DMADEVICES=y
CONFIG_INTEL_IOATDMA=mCONFIG_DMA_ENGINE=y
CONFIG_NET_DMA=yCONFIG_DCA=mCONFIG_VIRTIO=yCONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=yCONFIG_VIRTIO_BALLOON=yCONFIG_VIRTIO_MMIO=mCONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_STAGING=yCONFIG_ZRAM=mCONFIG_ZRAM_DEBUG=yCONFIG_ZCACHE=yCONFIG_ZSMALLOC=y
CONFIG_USB_WPAN_HCD=mCONFIG_X86_PLATFORM_DEVICES=yCONFIG_THINKPAD_ACPI=mCONFIG_THINKPAD_ACPI_ALSA_SUPPORT=yCONFIG_THINKPAD_ACPI_DEBUG=yCONFIG_THINKPAD_ACPI_VIDEO=yCONFIG_SENSORS_HDAPS=yCONFIG_ACPI_WMI=mCONFIG_ACPI_TOSHIBA=mCONFIG_INTEL_IPS=m
CONFIG_CLKEVT_I8253=yCONFIG_I8253_LOCK=yCONFIG_CLKBLD_I8253=yCONFIG_IOMMU_SUPPORT=y
CONFIG_VIRT_DRIVERS=yCONFIG_PM_DEVFREQ=y
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=yCONFIG_DEVFREQ_GOV_PERFORMANCE=yCONFIG_DEVFREQ_GOV_POWERSAVE=yCONFIG_DEVFREQ_GOV_USERSPACE=y
CONFIG_EDD=mCONFIG_FIRMWARE_MEMMAP=yCONFIG_EFI_VARS=mCONFIG_DMIID=yCONFIG_DMI_SYSFS=m
CONFIG_DCACHE_WORD_ACCESS=yCONFIG_EXT2_FS=yCONFIG_EXT2_FS_XATTR=yCONFIG_EXT2_FS_POSIX_ACL=yCONFIG_EXT2_FS_SECURITY=yCONFIG_EXT3_FS=yCONFIG_EXT3_DEFAULTS_TO_ORDERED=yCONFIG_EXT3_FS_XATTR=yCONFIG_EXT3_FS_POSIX_ACL=yCONFIG_EXT3_FS_SECURITY=yCONFIG_EXT4_FS=yCONFIG_EXT4_FS_XATTR=yCONFIG_EXT4_FS_POSIX_ACL=yCONFIG_EXT4_FS_SECURITY=yCONFIG_JBD=yCONFIG_JBD2=yCONFIG_FS_MBCACHE=yCONFIG_REISERFS_FS=mCONFIG_REISERFS_PROC_INFO=yCONFIG_REISERFS_FS_XATTR=yCONFIG_REISERFS_FS_POSIX_ACL=yCONFIG_REISERFS_FS_SECURITY=yCONFIG_XFS_FS=mCONFIG_XFS_QUOTA=yCONFIG_XFS_POSIX_ACL=yCONFIG_XFS_RT=yCONFIG_BTRFS_FS=mCONFIG_BTRFS_FS_POSIX_ACL=yCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_FS_POSIX_ACL=yCONFIG_EXPORTFS=mCONFIG_FILE_LOCKING=yCONFIG_FSNOTIFY=yCONFIG_DNOTIFY=yCONFIG_INOTIFY_USER=yCONFIG_FANOTIFY=yCONFIG_FANOTIFY_ACCESS_PERMISSIONS=yCONFIG_QUOTACTL=yCONFIG_QUOTACTL_COMPAT=yCONFIG_AUTOFS4_FS=mCONFIG_FUSE_FS=mCONFIG_CUSE=mCONFIG_GENERIC_ACL=y
CONFIG_FSCACHE=mCONFIG_FSCACHE_STATS=yCONFIG_FSCACHE_HISTOGRAM=yCONFIG_FSCACHE_DEBUG=yCONFIG_CACHEFILES=mCONFIG_CACHEFILES_DEBUG=y
CONFIG_ISO9660_FS=mCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_UDF_FS=mCONFIG_UDF_NLS=y
CONFIG_FAT_FS=mCONFIG_VFAT_FS=mCONFIG_FAT_DEFAULT_CODEPAGE=852CONFIG_FAT_DEFAULT_IOCHARSET="cp1250"CONFIG_NTFS_FS=mCONFIG_NTFS_RW=y
CONFIG_PROC_FS=yCONFIG_PROC_KCORE=yCONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=yCONFIG_SYSFS=yCONFIG_TMPFS=yCONFIG_TMPFS_POSIX_ACL=yCONFIG_TMPFS_XATTR=yCONFIG_HUGETLBFS=yCONFIG_HUGETLB_PAGE=yCONFIG_CONFIGFS_FS=m CONFIG_MISC_FILESYSTEMS=yCONFIG_JFFS2_FS=mCONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=yCONFIG_JFFS2_LZO=yCONFIG_JFFS2_RTIME=yCONFIG_JFFS2_RUBIN=y CONFIG_JFFS2_CMODE_PRIORITY=yCONFIG_CRAMFS=mCONFIG_SQUASHFS=mCONFIG_SQUASHFS_ZLIB=yCONFIG_SQUASHFS_XZ=y CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=mCONFIG_NFS_V2=yCONFIG_NFS_V3=yCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=yCONFIG_NFS_V4_1=yCONFIG_PNFS_FILE_LAYOUT=mCONFIG_PNFS_BLOCK=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" CONFIG_NFS_USE_KERNEL_DNS=yCONFIG_NFSD=mCONFIG_NFSD_V2_ACL=yCONFIG_NFSD_V3=yCONFIG_NFSD_V3_ACL=yCONFIG_NFSD_V4=yCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=mCONFIG_NFS_COMMON=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=yCONFIG_CIFS=mCONFIG_CIFS_XATTR=yCONFIG_CIFS_POSIX=yCONFIG_NLS=yCONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_852=m CONFIG_NLS_CODEPAGE_1250=mCONFIG_NLS_ASCII=yCONFIG_NLS_ISO8859_1=mCONFIG_NLS_ISO8859_2=mCONFIG_NLS_UTF8=mCONFIG_DLM=m
CONFIG_TRACE_IRQFLAGS_SUPPORT=yCONFIG_PRINTK_TIME=yCONFIG_DEFAULT_MESSAGE_LOGLEVEL=4CONFIG_ENABLE_WARN_DEPRECATED=yCONFIG_ENABLE_MUST_CHECK=yCONFIG_FRAME_WARN=2048CONFIG_MAGIC_SYSRQ=yCONFIG_DEBUG_FS=yCONFIG_DEBUG_KERNEL=yCONFIG_LOCKUP_DETECTOR=yCONFIG_HARDLOCKUP_DETECTOR=yCONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0CONFIG_PANIC_ON_OOPS_VALUE=0CONFIG_DETECT_HUNG_TASK=yCONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0CONFIG_SCHED_DEBUG=yCONFIG_SCHEDSTATS=yCONFIG_TIMER_STATS=yCONFIG_SLUB_STATS=yCONFIG_DEBUG_KMEMLEAK=yCONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=yCONFIG_DEBUG_PREEMPT=yCONFIG_DEBUG_SPINLOCK=yCONFIG_DEBUG_MUTEXES=yCONFIG_DEBUG_LOCK_ALLOC=yCONFIG_PROVE_LOCKING=yCONFIG_PROVE_RCU=yCONFIG_LOCKDEP=yCONFIG_LOCK_STAT=yCONFIG_TRACE_IRQFLAGS=yCONFIG_DEBUG_ATOMIC_SLEEP=yCONFIG_STACKTRACE=yCONFIG_DEBUG_BUGVERBOSE=yCONFIG_DEBUG_INFO=yCONFIG_DEBUG_INFO_REDUCED=yCONFIG_DEBUG_MEMORY_INIT=yCONFIG_DEBUG_NOTIFIERS=yCONFIG_ARCH_WANT_FRAME_POINTERS=yCONFIG_FRAME_POINTER=yCONFIG_RCU_CPU_STALL_TIMEOUT=60CONFIG_RCU_CPU_STALL_VERBOSE=yCONFIG_RCU_CPU_STALL_INFO=yCONFIG_RCU_TRACE=yCONFIG_LKDTM=mCONFIG_LATENCYTOP=yCONFIG_USER_STACKTRACE_SUPPORT=yCONFIG_NOP_TRACER=yCONFIG_HAVE_FUNCTION_TRACER=yCONFIG_HAVE_FUNCTION_GRAPH_TRACER=yCONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=yCONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=yCONFIG_HAVE_DYNAMIC_FTRACE=yCONFIG_HAVE_FTRACE_MCOUNT_RECORD=yCONFIG_HAVE_SYSCALL_TRACEPOINTS=yCONFIG_HAVE_C_RECORDMCOUNT=yCONFIG_TRACER_MAX_TRACE=yCONFIG_RING_BUFFER=yCONFIG_EVENT_TRACING=yCONFIG_CONTEXT_SWITCH_TRACER=yCONFIG_RING_BUFFER_ALLOW_SWAP=yCONFIG_TRACING=yCONFIG_GENERIC_TRACER=yCONFIG_TRACING_SUPPORT=yCONFIG_FTRACE=yCONFIG_FUNCTION_TRACER=yCONFIG_FUNCTION_GRAPH_TRACER=yCONFIG_IRQSOFF_TRACER=yCONFIG_PREEMPT_TRACER=yCONFIG_SCHED_TRACER=yCONFIG_FTRACE_SYSCALLS=yCONFIG_BRANCH_PROFILE_NONE=yCONFIG_STACK_TRACER=yCONFIG_BLK_DEV_IO_TRACE=yCONFIG_KPROBE_EVENT=yCONFIG_UPROBE_EVENT=yCONFIG_PROBE_EVENTS=yCONFIG_DYNAMIC_FTRACE=yCONFIG_FUNCTION_PROFILER=yCONFIG_FTRACE_MCOUNT_RECORD=yCONFIG_MMIOTRACE=yCONFIG_MMIOTRACE_TEST=mCONFIG_DYNAMIC_DEBUG=yCONFIG_DMA_API_DEBUG=yCONFIG_HAVE_ARCH_KGDB=yCONFIG_HAVE_ARCH_KMEMCHECK=yCONFIG_X86_VERBOSE_BOOTUP=yCONFIG_EARLY_PRINTK=yCONFIG_DEBUG_RODATA=yCONFIG_HAVE_MMIOTRACE_SUPPORT=yCONFIG_IO_DELAY_TYPE_0X80=0CONFIG_IO_DELAY_TYPE_0XED=1CONFIG_IO_DELAY_TYPE_UDELAY=2CONFIG_IO_DELAY_TYPE_NONE=3CONFIG_IO_DELAY_0X80=yCONFIG_DEFAULT_IO_DELAY_TYPE=0CONFIG_DEBUG_BOOT_PARAMS=yCONFIG_OPTIMIZE_INLINING=y
CONFIG_KEYS=yCONFIG_ENCRYPTED_KEYS=yCONFIG_SECURITY=yCONFIG_SECURITY_NETWORK=yCONFIG_LSM_MMAP_MIN_ADDR=65535CONFIG_SECURITY_SELINUX=yCONFIG_SECURITY_SELINUX_BOOTPARAM=yCONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1CONFIG_SECURITY_SELINUX_DISABLE=yCONFIG_SECURITY_SELINUX_DEVELOP=yCONFIG_SECURITY_SELINUX_AVC_STATS=yCONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1CONFIG_INTEGRITY=yCONFIG_EVM=yCONFIG_DEFAULT_SECURITY_SELINUX=yCONFIG_DEFAULT_SECURITY="selinux"CONFIG_XOR_BLOCKS=mCONFIG_ASYNC_CORE=mCONFIG_ASYNC_MEMCPY=mCONFIG_ASYNC_XOR=mCONFIG_ASYNC_PQ=mCONFIG_ASYNC_RAID6_RECOV=mCONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=yCONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=yCONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=yCONFIG_CRYPTO_ALGAPI2=yCONFIG_CRYPTO_AEAD=mCONFIG_CRYPTO_AEAD2=yCONFIG_CRYPTO_BLKCIPHER=yCONFIG_CRYPTO_BLKCIPHER2=yCONFIG_CRYPTO_HASH=yCONFIG_CRYPTO_HASH2=yCONFIG_CRYPTO_RNG=yCONFIG_CRYPTO_RNG2=yCONFIG_CRYPTO_PCOMP2=yCONFIG_CRYPTO_MANAGER=yCONFIG_CRYPTO_MANAGER2=yCONFIG_CRYPTO_USER=mCONFIG_CRYPTO_MANAGER_DISABLE_TESTS=yCONFIG_CRYPTO_GF128MUL=mCONFIG_CRYPTO_PCRYPT=mCONFIG_CRYPTO_WORKQUEUE=yCONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_CBC=yCONFIG_CRYPTO_ECB=mCONFIG_CRYPTO_LRW=mCONFIG_CRYPTO_XTS=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_CRC32C=yCONFIG_CRYPTO_MD4=mCONFIG_CRYPTO_MD5=yCONFIG_CRYPTO_MICHAEL_MIC=mCONFIG_CRYPTO_SHA1=yCONFIG_CRYPTO_SHA1_SSSE3=yCONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_AES=yCONFIG_CRYPTO_AES_X86_64=mCONFIG_CRYPTO_AES_NI_INTEL=mCONFIG_CRYPTO_ARC4=mCONFIG_CRYPTO_BLOWFISH_COMMON=mCONFIG_CRYPTO_BLOWFISH_X86_64=mCONFIG_CRYPTO_DES=mCONFIG_CRYPTO_TWOFISH_COMMON=mCONFIG_CRYPTO_TWOFISH_X86_64=mCONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_HW=yCONFIG_HAVE_KVM=yCONFIG_HAVE_KVM_IRQCHIP=yCONFIG_HAVE_KVM_EVENTFD=yCONFIG_KVM_APIC_ARCHITECTURE=yCONFIG_KVM_MMIO=yCONFIG_KVM_ASYNC_PF=yCONFIG_HAVE_KVM_MSI=yCONFIG_VIRTUALIZATION=yCONFIG_KVM=mCONFIG_KVM_INTEL=mCONFIG_KVM_AMD=mCONFIG_KVM_MMU_AUDIT=yCONFIG_VHOST_NET=mCONFIG_BINARY_PRINTF=y
CONFIG_RAID6_PQ=mCONFIG_BITREVERSE=yCONFIG_GENERIC_STRNCPY_FROM_USER=yCONFIG_GENERIC_STRNLEN_USER=yCONFIG_GENERIC_FIND_FIRST_BIT=yCONFIG_GENERIC_PCI_IOMAP=yCONFIG_GENERIC_IOMAP=yCONFIG_GENERIC_IO=yCONFIG_CRC_CCITT=mCONFIG_CRC16=yCONFIG_CRC_T10DIF=yCONFIG_CRC_ITU_T=mCONFIG_CRC32=yCONFIG_CRC32_SLICEBY8=yCONFIG_LIBCRC32C=mCONFIG_ZLIB_INFLATE=yCONFIG_ZLIB_DEFLATE=mCONFIG_LZO_COMPRESS=yCONFIG_LZO_DECOMPRESS=yCONFIG_XZ_DEC=yCONFIG_XZ_DEC_X86=yCONFIG_XZ_DEC_POWERPC=yCONFIG_XZ_DEC_IA64=yCONFIG_XZ_DEC_ARM=yCONFIG_XZ_DEC_ARMTHUMB=yCONFIG_XZ_DEC_SPARC=yCONFIG_XZ_DEC_BCJ=yCONFIG_DECOMPRESS_GZIP=yCONFIG_DECOMPRESS_BZIP2=yCONFIG_DECOMPRESS_LZMA=yCONFIG_DECOMPRESS_XZ=yCONFIG_DECOMPRESS_LZO=yCONFIG_TEXTSEARCH=yCONFIG_TEXTSEARCH_KMP=mCONFIG_TEXTSEARCH_BM=mCONFIG_TEXTSEARCH_FSM=mCONFIG_HAS_IOMEM=yCONFIG_HAS_IOPORT=yCONFIG_HAS_DMA=yCONFIG_CHECK_SIGNATURE=yCONFIG_CPU_RMAP=yCONFIG_DQL=yCONFIG_NLATTR=yCONFIG_LRU_CACHE=mCONFIG_AVERAGE=yCONFIG_DDR=y
Quoted 5 lines · Zdenek Kabelac
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.
Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 4 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLE
But this leaves me very puzzled.
Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?
I see no mention of madvise or MADV_REMOVE or fallocate or anything related in that git tree.
If you have something else running at the same time, which happens to use madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.
I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.
I'm not an LVM user, so I doubt I'll be able to reproduce your setup.
Any ideas from the DM guys? Has anyone else seen anything like this?
Do all your oopses look like this one?
Thanks,
Hugh
Quoted 112 lines
My HW - t61, 4GB
Zdenek
BUG: unable to handle kernel NULL pointer dereference at 0000000000000178 IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] PGD 0 Oops: 0000 [#1] PREEMPT SMP CPU 1 Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq
async_xorxorasync_txraid6_pqraid1md_modreiserfs xfsdm_mirrordm_region_hashdm_logdm_snapshotip6_tablesebtable_natebtablesipt_MASQUERADEiptable_natnf_natnf_conntrack_ipv4nf_defrag_ipv4xt_statenf_conntrackipt_REJECTiptable_manglext_tcpudpiptable_filterip_tablesx_tablesbridge stp llc ipv6snd_hda_codec_analogarc4hid_genericiwl3945 iwlegacy microcode psmouseserio_rawmac80211i2c_i801i2c_coresnd_hda_intelsnd_hda_codeccfg80211lpc_ichmfd_coresnd_pcmsnd_page_alloce1000esnd_timerwmithinkpad_acpinvram snd soundcore evdev usbhid hidbinfmt_miscloopvhost_nettunkvm_intelnfsd kvm exportfsnfs_aclauth_rpcgsslockd sunrpcdm_modautofs4 pcmciasr_modcdromsdhci_pcisdhcimmc_coreyenta_socketehci_hcduhci_hcdusbcoreusb_commonvideo backlightPid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO
RIP: 0010:[<ffffffffa031204d>] [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP: 0018:ffff880126b77c68 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607 RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40 RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400 R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task ffff8801344ca310) Stack: ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0 Call Trace: [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror] [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror] [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod] [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod] [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod] [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod] [<ffffffff81060065>] process_one_work+0x1a5/0x6b0 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod] [<ffffffff810608ee>] worker_thread+0x12e/0x2d0 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210 [<ffffffff81069cf2>] kthread+0xa2/0xb0 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0 [<ffffffff81533330>] ? gs_change+0xb/0xb Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP [<ffffffffa031204d>]dm_rh_region_context+0xd/0x20[dm_region_hash]
RSP <ffff880126b77c68>
CR2: 0000000000000178# bad: [
cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' ofgit://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new genericstrnlen_user()function
git bisect good5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' ofgit://git.kernel.dk/linux-block
git bisect bada70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' ofgit://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' ofgit://git.linaro.org/people/rmk/linux-arm
git bisect bad227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dumpPG_compound_lockpage flag
git bisect badbe9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible
git bisect good1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729]mm/vmstat.c: remove debug fs entries on failure of file creation and madeextfrag_debug_rootdentry local
git bisect goodbde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem
git bisect good549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone
git bisect goodbde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: removetruncate_range
git bisect bad17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing
git bisect goodec9516fbc5fa814014991e1ae7f8860127122105
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 11 lines · Zdenek Kabelac
Quoted 5 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.
Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?
Yes - that's it -
make
as root:
cd test
make check_local
(inside test subdirectory should be enough, if not - just report any problem)
Quoted 9 lines
I see no mention of madvise or
MADV_REMOVEor fallocate or anything related in that git tree.If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE)on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.
What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB)
Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.
So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.
Quoted 3 lines
I'll be surprised if that is the case, but it's something that you can easily check by inserting a
WARN_ON(1)inmm/madvise.cmadvise_remove():
that should tell us what process is using it.
I could try that if that will help.
Quoted 1 line
I'm not an LVM user, so I doubt I'll be able to reproduce your setup.
Shouldn't be hard to run - unsure if every config setup is influnenced or just mine config.
Quoted 4 lines
Any ideas from the DM guys? Has anyone else seen anything like this?
Do all your oopses look like this one?
I think I've get yet another one - but also within dm_rh_region
It could be that your patch exposed problem of some different part of stack - not really sure - it's just now with 3.5 this crash will not allow to pass whole test suite - I've tried also in kvm machine and it's been reproducible (so in the worst case I could eventually send you 2GB image)
The problem is - there is not a 'single test case' to trigger the oops (at least I've not figured out one) - it's the combination of multiple tests running after each other - but for simplication this should be enough:
make check_local T=shell/lvconvert
Which usually dies on shell/lvconvert-repair-transient.sh
Zdenek
Quoted 41 lines · Zdenek Kabelac
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 6 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?
Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 9 lines
I see no mention of madvise or
MADV_REMOVEor fallocate or anything related in that git tree.If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE)on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.
Quoted 6 lines
What the test is doing - it creates file in
LVM_TEST_DIR(default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB)Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.
This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.
Quoted 3 lines
So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.
What filesystem do you have for /tmp?
If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".
The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.
Quoted 6 lines
Quoted 4 lines
I'll be surprised if that is the case, but it's something that you can easily check by inserting a
WARN_ON(1)inmm/madvise.cmadvise_remove():
that should tell us what process is using it.I could try that if that will help.
That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before.
Quoted 5 lines
Quoted 2 lines
I'm not an LVM user, so I doubt I'll be able to reproduce your setup.
Shouldn't be hard to run - unsure if every config setup is influnenced or just mine config.
I'll start from your config.
Quoted 13 lines
Quoted 5 lines
Any ideas from the DM guys? Has anyone else seen anything like this?
Do all your oopses look like this one?
I think I've get yet another one - but also within
dm_rh_regionIt could be that your patch exposed problem of some different part of stack - not really sure - it's just now with 3.5 this crash will not allow to pass whole test suite - I've tried also in kvm machine and it's been reproducible (so in the worst case I could eventually send you 2GB image)
Quoted 7 lines
The problem is - there is not a 'single test case' to trigger the oops (at least I've not figured out one) - it's the combination of multiple tests running after each other - but for simplication this should be enough:
make
check_localT=shell/lvconvertWhich usually dies on
shell/lvconvert-repair-transient.sh
Thanks again, I'll report back later.
Hugh
Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
Quoted 66 lines · Zdenek Kabelac
Quoted 37 lines
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 6 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?
Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 5 lines
If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE)on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.Quoted 6 lines
What the test is doing - it creates file in
LVM_TEST_DIR(default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB)Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.
This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.Quoted 3 lines
So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.
What filesystem do you have for /tmp?
If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".
The commit before,
83e4fa9c16e4af7122e31be3eca5d57881d236fe"tmpfs: support fallocateFALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.
Quoted 3 lines
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever,
madvise(,,MADV_REMOVE)should be completely irrelevant.
While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)
So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
3.1 is the first kernel which does survive (checked 5 repeated runs)
And you are correct, the first commit which causes crash really is83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'make check_local' execution - and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in the first pass.
I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.
So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?
Quoted 11 lines
Quoted 7 lines
Quoted 5 lines
I'll be surprised if that is the case, but it's something that you can easily check by inserting a
WARN_ON(1)inmm/madvise.cmadvise_remove():
that should tell us what process is using it.I could try that if that will help.
That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before.
the only thing which could be tricky is udev support
(by default it's not enabled ---enable-udev_sync)
However Debian based distros are distributing their own rules, which are not 100% compatible with upstream and create some unpredictable issues, where the slowness is the least problem.)
If you have Fedora Rawhide with latest lvm2 installed - you should get pretty well configured system for running test dir (unfortunately there is no way to virtualize udev...)
Zdenek
Quoted 49 lines · Zdenek Kabelac
Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
Quoted 48 lines · Zdenek Kabelac
Quoted 47 lines
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 16 lines · Zdenek Kabelac
Quoted 8 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17Quoted 7 lines
Quoted 1 line
mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 9 lines
If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE)on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.What the test is doing - it creates file in
LVM_TEST_DIR(default is /tmp)
I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.
And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards.
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
and using loop device to simulate device (small size - it should fit bellow
200MB)Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.
This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.
Sorry, I never reached it yesterday, but arrived there this morning.
Quoted 26 lines
Quoted 6 lines
Quoted 4 lines
So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.
What filesystem do you have for /tmp?
From your later remarks, I inferred tmpfs.
Quoted 13 lines
If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".
The commit before,
83e4fa9c16e4af7122e31be3eca5d57881d236fe"tmpfs: support fallocateFALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.Quoted 3 lines
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever,
madvise(,,MADV_REMOVE)should be completely irrelevant.
Quoted 8 lines
While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)
So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
3.1 is the first kernel which does survive (checked 5 repeated runs)
Very useful research, thank you.
Quoted 10 lines
And you are correct, the first commit which causes crash really is
83e4fa9c16e4afwhen I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'makecheck_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit3f31d07571ethe crash always happens in the first pass.
Quoted 5 lines
I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.
So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?
No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.
Quoted 20 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
I'll be surprised if that is the case, but it's something that you can easily check by inserting a
WARN_ON(1)inmm/madvise.cmadvise_remove():
that should tell us what process is using it.Quoted 5 lines
Quoted 1 line
I could try that if that will help.
That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before.
the only thing which could be tricky is udev support
(by default it's not enabled ---enable-udev_sync)
However Debian based distros are distributing their own rules, which are not 100% compatible with upstream and create some unpredictable issues, where the slowness is the least problem.)
Quoted 3 lines
If you have Fedora Rawhide with latest lvm2 installed - you should get pretty
well configured system for running test dir (unfortunately there is no way
to virtualize udev...)
I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io().
But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.
I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.)
I've added a few people to the Cc, innocents fingered by git blame on some REQ_DISCARD lines in drivers/md. But I'll back out myself:
I'm not short of other things to look at, and have little to offer in drivers/md.
Hugh
Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
Quoted 121 lines · Zdenek Kabelac
Quoted 49 lines
Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
Quoted 48 lines · Zdenek Kabelac
Quoted 47 lines
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 16 lines · Zdenek Kabelac
Quoted 8 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17Quoted 7 lines
Quoted 1 line
mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 9 lines
If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE)on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.What the test is doing - it creates file in
LVM_TEST_DIR(default is /tmp)I ran "LVM_TEST_DIR=/tmp make
check_local":
without that it appeared to be using a subdirectory made under test/.And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack
lib/config/defaults.hto #defineDEFAULT_ISSUE_DISCARDS1 after I spotted a warning message aboutissue_discards.Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
and using loop device to simulate device (small size - it should fit bellow
200MB)Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.
This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.Sorry, I never reached it yesterday, but arrived there this morning.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.
What filesystem do you have for /tmp?
From your later remarks, I inferred tmpfs.
Quoted 17 lines
Quoted 13 lines
If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".
The commit before,
83e4fa9c16e4af7122e31be3eca5d57881d236fe"tmpfs: support fallocateFALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.Quoted 3 lines
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever,
madvise(,,MADV_REMOVE)should be completely irrelevant.Quoted 8 lines
While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)
So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
3.1 is the first kernel which does survive (checked 5 repeated runs)
Very useful research, thank you.
Quoted 13 lines
And you are correct, the first commit which causes crash really is
83e4fa9c16e4afwhen I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'makecheck_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit3f31d07571ethe crash always happens in the first pass.I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.
Quoted 6 lines
Quoted 2 lines
So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?
No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into
drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.
Well I've played meanwhile the game with minimized kernel config and the outcome is:
last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef
dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices
Quoted 3 lines
I don't have Fedora Rawhide on, but after hacking
ISSUE_DISCARDSI did quickly crash around where you expected; though in my case it was indm_rh_dec()called frommirror_end_io().
Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.
Quoted 9 lines
But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.
I think it's fair to assume that there's something wrong with DM's handling of
REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications toREQ_DISCARDhandling in the block layer, it's been rather troublesome.)
So does anyone has some idea what should be checked next ?
Zdenek
Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
Quoted 149 lines
Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
Quoted 121 lines · Zdenek Kabelac
Quoted 49 lines
Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
Quoted 48 lines · Zdenek Kabelac
Quoted 47 lines
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 16 lines · Zdenek Kabelac
Quoted 8 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17Quoted 7 lines
Quoted 1 line
mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 9 lines
If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE)on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.What the test is doing - it creates file in
LVM_TEST_DIR(default is /tmp)I ran "LVM_TEST_DIR=/tmp make
check_local":
without that it appeared to be using a subdirectory made under test/.And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack
lib/config/defaults.hto #defineDEFAULT_ISSUE_DISCARDS1 after I spotted a warning message aboutissue_discards.Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
and using loop device to simulate device (small size - it should fit bellow
200MB)Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.
This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.Sorry, I never reached it yesterday, but arrived there this morning.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.
What filesystem do you have for /tmp?
From your later remarks, I inferred tmpfs.
Quoted 17 lines
Quoted 13 lines
If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".
The commit before,
83e4fa9c16e4af7122e31be3eca5d57881d236fe"tmpfs: support fallocateFALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.Quoted 3 lines
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever,
madvise(,,MADV_REMOVE)should be completely irrelevant.Quoted 8 lines
While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)
So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
3.1 is the first kernel which does survive (checked 5 repeated runs)
Very useful research, thank you.
Quoted 13 lines
And you are correct, the first commit which causes crash really is
83e4fa9c16e4afwhen I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'makecheck_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit3f31d07571ethe crash always happens in the first pass.I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.
Quoted 6 lines
Quoted 2 lines
So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?
No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into
drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.Well I've played meanwhile the game with minimized kernel config and the outcome is:
last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef
dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devicesQuoted 3 lines
I don't have Fedora Rawhide on, but after hacking
ISSUE_DISCARDSI did quickly crash around where you expected; though in my case it was indm_rh_dec()called frommirror_end_io().Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.
Quoted 4 lines
But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.
Quoted 7 lines
Quoted 5 lines
I think it's fair to assume that there's something wrong with DM's handling of
REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications toREQ_DISCARDhandling in the block layer, it's been rather troublesome.)So does anyone has some idea what should be checked next ?
I've enabled more kernel debugging tools - and this came out when running test suite:
Unsure if this could be source of troubles?
kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1
Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4
Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 6464CTO/6464CTO
RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090) Stack: ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 Call Trace: [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] [<ffffffff81050177>] process_one_work+0x197/0x460 [<ffffffff81050116>] ? process_one_work+0x136/0x460 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] [<ffffffff81050986>] worker_thread+0x126/0x2d0 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 [<ffffffff81056066>] kthread+0x96/0xa0 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8133b610>] ? gs_change+0xb/0xb Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP <ffff880136e41aa0> ---[ end trace 040dfd9d68fccabc ]---
Zdenek
Quoted 10 lines · Zdenek Kabelac
Date: Mon, 02 Jul 2012 00:03:07 +0200
From: Zdenek Kabelac <[email protected]>
To: Hugh Dickins <[email protected]>
Cc: Mike Snitzer <[email protected]>, Mikulas Patocka <[email protected]>,
Joe Thornber <[email protected]>,
LVM general discussion and development <[email protected]>, [email protected], Alasdair G Kergon <[email protected]>, [email protected], Lukas Czerner <[email protected]>
Subject: Re: Regression withFALLOC_FL_PUNCH_HOLEin 3.5-rc kernel
Quoted 174 lines
Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
Quoted 173 lines
Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
Quoted 139 lines · Zdenek Kabelac
Quoted 54 lines
Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
Quoted 53 lines · Zdenek Kabelac
Quoted 52 lines
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 18 lines · Zdenek Kabelac
Quoted 10 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit
this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does
not
included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17Quoted 7 lines
Quoted 1 line
mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 11 lines
If you have something else running at the same time, which happens to
usemadvise(,,MADV_REMOVE)on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.What the test is doing - it creates file in
LVM_TEST_DIR(default is
/tmp)I ran "LVM_TEST_DIR=/tmp make
check_local":
without that it appeared to be using a subdirectory made under test/.And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack
lib/config/defaults.hto #defineDEFAULT_ISSUE_DISCARDS1 after I spotted a warning message aboutissue_discards.Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
and using loop device to simulate device (small size - it should fit bellow
200MB)Within this file second layer through virtual DM devices is created and
simulates various numbers of PV devices to play with.This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.Sorry, I never reached it yesterday, but arrived there this morning.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
So since everything now support TRIM - such operations should be passed
down to the backend file - which probably triggers the path.What filesystem do you have for /tmp?
From your later remarks, I inferred tmpfs.
Quoted 25 lines
Quoted 15 lines
If tmpfs, then it will make much more sense if we assume your bisection
endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might
have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".The commit before,
83e4fa9c16e4af7122e31be3eca5d57881d236fe"tmpfs: support fallocateFALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.Quoted 3 lines
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever,
madvise(,,MADV_REMOVE)should be completely irrelevant.While I've been aware of the fact that tmpfs was enhanced with trim support -
I've not tried to run on real ext4 filesystem since for my tests I'm using
tmpfs for quite some time to safe rewrites of SSD :)Quoted 4 lines
So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
3.1 is the first kernel which does survive (checked 5 repeated runs)
Very useful research, thank you.
Quoted 20 lines
And you are correct, the first commit which causes crash really is
83e4fa9c16e4afwhen I use tmpfs as backend storage - the problem why I've
missed to properly identify this commit in my bisect is that crash usually
happens on the second pass of the lvm test suite 'makecheck_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on
3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit3f31d07571ethe crash always happens in
the first pass.I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
and it's crashing as well - so it's probably not uniqueness of my config.Quoted 7 lines
Quoted 3 lines
So is there any primary suspect in 3.2 which is worth to check - or I need
another day to play another bisect game ?No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into
drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.Well I've played meanwhile the game with minimized kernel config and the outcome is:
last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef
dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devicesQuoted 3 lines
I don't have Fedora Rawhide on, but after hacking
ISSUE_DISCARDSI did quickly crash around where you expected; though in my case it was indm_rh_dec()called frommirror_end_io().Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.
Quoted 9 lines
But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.
I think it's fair to assume that there's something wrong with DM's handling of
REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications toREQ_DISCARDhandling in the block layer, it's been rather troublesome.)
Hi Zdenek,
Quoted 2 lines
Quoted 2 lines
So does anyone has some idea what should be checked next ?
I've missed the information in the thread, but since you're the most familiar with the lvm test suite, could you distil the reproducer which does not require to run lvm test suit ?
Quoted 7 lines
I've enabled more kernel debugging tools - and this came out when running test suite:
Unsure if this could be source of troubles?
Quoted 8 lines
kernel BUG at
arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMPDEBUG_PAGEALLOC
CPU 1Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stpllc psmouse
serio_rawcryptomgr aead arc4crypto_algapisr_modcdrom iwl3945iwl_legacymac80211 cfg80211 crypto e1000e evdev loopdm_modautofs4Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO
So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ?
Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ?
Looking at git log, this commit has been brought to my attention:
0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support
seems related to this crash.
Please retest with recent kernel.
Thanks!
-Lukas
Quoted 53 lines
6464CTO/6464CTO
RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090) Stack: ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 Call Trace: [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] [<ffffffff81050177>] process_one_work+0x197/0x460 [<ffffffff81050116>] ? process_one_work+0x136/0x460 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] [<ffffffff81050986>] worker_thread+0x126/0x2d0 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 [<ffffffff81056066>] kthread+0x96/0xa0 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8133b610>] ? gs_change+0xb/0xb Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 0078 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP [<ffffffff81021f00>]__phys_addr+0x60/0x70
RSP <ffff880136e41aa0>---[ end trace 040dfd9d68fccabc ]---Zdenek
Quoted 235 lines · Lukᅵ Czerner
Date: Mon, 2 Jul 2012 11:00:06 +0200 (CEST)
From: Lukᅵ Czerner <[email protected]>
To: Zdenek Kabelac <[email protected]>
Cc: Hugh Dickins <[email protected]>, Mike Snitzer <[email protected]>,
Mikulas Patocka <[email protected]>, Joe Thornber <[email protected]>, LVM general discussion and development <[email protected]>, [email protected], Alasdair G Kergon <[email protected]>, [email protected], Lukas Czerner <[email protected]>
Subject: Re: Regression withFALLOC_FL_PUNCH_HOLEin 3.5-rc kernelQuoted 10 lines · Zdenek Kabelac
Date: Mon, 02 Jul 2012 00:03:07 +0200
From: Zdenek Kabelac <[email protected]>
To: Hugh Dickins <[email protected]>
Cc: Mike Snitzer <[email protected]>, Mikulas Patocka <[email protected]>,
Joe Thornber <[email protected]>,
LVM general discussion and development <[email protected]>, [email protected], Alasdair G Kergon <[email protected]>, [email protected], Lukas Czerner <[email protected]>
Subject: Re: Regression withFALLOC_FL_PUNCH_HOLEin 3.5-rc kernelQuoted 174 lines
Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
Quoted 173 lines
Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
Quoted 139 lines · Zdenek Kabelac
Quoted 54 lines
Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
Quoted 53 lines · Zdenek Kabelac
Quoted 52 lines
Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
Quoted 18 lines · Zdenek Kabelac
Quoted 10 lines
When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit
this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does
not
included anything related to this oops I've went for bisect.Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.
Quoted 3 lines
Game result is commit:
3f31d07571eeea18a7d34db9af21d2285b807a17Quoted 7 lines
Quoted 1 line
mm/fs: route
MADV_REMOVEtoFALLOC_FL_PUNCH_HOLEBut this leaves me very puzzled.
Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?Yes - that's it -
make
as root:
cd test
makecheck_local(inside test subdirectory should be enough, if not - just report any problem)
Quoted 11 lines
If you have something else running at the same time, which happens to
usemadvise(,,MADV_REMOVE)on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting thedm_regionin your oops.What the test is doing - it creates file in
LVM_TEST_DIR(default is
/tmp)I ran "LVM_TEST_DIR=/tmp make
check_local":
without that it appeared to be using a subdirectory made under test/.And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack
lib/config/defaults.hto #defineDEFAULT_ISSUE_DISCARDS1 after I spotted a warning message aboutissue_discards.Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
and using loop device to simulate device (small size - it should fit bellow
200MB)Within this file second layer through virtual DM devices is created and
simulates various numbers of PV devices to play with.This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.Sorry, I never reached it yesterday, but arrived there this morning.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
So since everything now support TRIM - such operations should be passed
down to the backend file - which probably triggers the path.What filesystem do you have for /tmp?
From your later remarks, I inferred tmpfs.
Quoted 25 lines
Quoted 15 lines
If tmpfs, then it will make much more sense if we assume your bisection
endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might
have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".The commit before,
83e4fa9c16e4af7122e31be3eca5d57881d236fe"tmpfs: support fallocateFALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.Quoted 3 lines
Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever,
madvise(,,MADV_REMOVE)should be completely irrelevant.While I've been aware of the fact that tmpfs was enhanced with trim support -
I've not tried to run on real ext4 filesystem since for my tests I'm using
tmpfs for quite some time to safe rewrites of SSD :)Quoted 4 lines
So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
3.1 is the first kernel which does survive (checked 5 repeated runs)
Very useful research, thank you.
Quoted 20 lines
And you are correct, the first commit which causes crash really is
83e4fa9c16e4afwhen I use tmpfs as backend storage - the problem why I've
missed to properly identify this commit in my bisect is that crash usually
happens on the second pass of the lvm test suite 'makecheck_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on
3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit3f31d07571ethe crash always happens in
the first pass.I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
and it's crashing as well - so it's probably not uniqueness of my config.Quoted 7 lines
Quoted 3 lines
So is there any primary suspect in 3.2 which is worth to check - or I need
another day to play another bisect game ?No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into
drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.Well I've played meanwhile the game with minimized kernel config and the outcome is:
last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef
dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devicesQuoted 3 lines
I don't have Fedora Rawhide on, but after hacking
ISSUE_DISCARDSI did quickly crash around where you expected; though in my case it was indm_rh_dec()called frommirror_end_io().Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.
Quoted 9 lines
But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.
I think it's fair to assume that there's something wrong with DM's handling of
REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications toREQ_DISCARDhandling in the block layer, it's been rather troublesome.)Hi Zdenek,
Quoted 2 lines
Quoted 2 lines
So does anyone has some idea what should be checked next ?
I've missed the information in the thread, but since you're the most familiar with the lvm test suite, could you distil the reproducer which does not require to run lvm test suit ?
Quoted 7 lines
I've enabled more kernel debugging tools - and this came out when running test suite:
Unsure if this could be source of troubles?
Quoted 8 lines
kernel BUG at
arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMPDEBUG_PAGEALLOC
CPU 1Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stpllc psmouse
serio_rawcryptomgr aead arc4crypto_algapisr_modcdrom iwl3945iwl_legacymac80211 cfg80211 crypto e1000e evdev loopdm_modautofs4Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO
So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ?
Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ?
Looking at git log, this commit has been brought to my attention:
Quoted 61 lines
0c535e0d6f463365c29623350dbd91642363c39bdm io: fix discard supportseems related to this crash.
Please retest with recent kernel.
Thanks!
-LukasQuoted 52 lines
6464CTO/6464CTO
RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090) Stack: ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 Call Trace: [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] [<ffffffff81050177>] process_one_work+0x197/0x460 [<ffffffff81050116>] ? process_one_work+0x136/0x460 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] [<ffffffff81050986>] worker_thread+0x126/0x2d0 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 [<ffffffff81056066>] kthread+0x96/0xa0 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8133b610>] ? gs_change+0xb/0xb Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 0078 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP [<ffffffff81021f00>]__phys_addr+0x60/0x70
RSP <ffff880136e41aa0>---[ end trace 040dfd9d68fccabc ]---Zdenek
So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
(https://lkml.org/lkml/2012/6/30/98) I think that this is problem in the device mapper itself. I do not think it has anything to do with tmpfs or mm. According to bisects from Zdenek it clearly shows that the problem appear when the discard support for the loop device is added, so it is most likely related to the dm discard support.
Anyway, the backtrace points to the NULL pointed dereference in dm_rh_region_context() which is simple function:
void *dm_rh_region_context(struct dm_region *reg)
{
return reg->rh->context;
}so either reg, or reg-rh is NULL. Now the only place this is used is from recovery_complete() in dm-raid1.c. So this is somewhat related to raid recovery. I am not familiar with the dm code, but can someone from the dm team look at this ?
But just to be sure to rule out the punch hole thing Zdenek can you run your tests on the "real" discard capable device ? Or at least on the device which does not convert discard requests into punch hole ? You can use scsi_debug to create such device:
modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1
Thanks!
-Lukas
Quoted 11 lines · Lukáš Czerner
Quoted 11 lines
So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ?
Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ?
Quoted 7 lines
Quoted 7 lines
Looking at git log, this commit has been brought to my attention:
0c535e0d6f463365c29623350dbd91642363c39bdm io: fix discard supportseems related to this crash.
Please retest with recent kernel.
Ah, you beat me to recommending that fix ;)
Quoted 6 lines
So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
(https://lkml.org/lkml/2012/6/30/98) I think that this is problem in the device mapper itself. I do not think it has anything to do with tmpfs or mm. According to bisects from Zdenek it clearly shows that the problem appear when the discard support for the loop device is added, so it is most likely related to the dm discard support.
What about using scsi_debug with the dm-mirror target?
Never say never, DM-mirror and/or dm-io code could still have an issue, but the commit referenced above did fix discard with the mirror target back in 3.3.
Quoted 4 lines
Anyway, the backtrace points to the NULL pointed dereference in
dm_rh_region_context()which is simple function:void *dm_rh_region_context(struct
dm_region*reg)
Quoted 8 lines
{ return reg->rh->context; }so either reg, or reg-rh is NULL. Now the only place this is used is from
recovery_complete()indm-raid1.c. So this is somewhat related to raid recovery. I am not familiar with the dm code, but can someone from the dm team look at this ?
I'll coordiinate with Zdenek.
Quoted 6 lines
But just to be sure to rule out the punch hole thing Zdenek can you run your tests on the "real" discard capable device ? Or at least on the device which does not convert discard requests into punch hole ? You can use
scsi_debugto create such device:modprobe
scsi_debugdev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1
Great minds think alike ;)
Not sure why you're testing such old kernels without regard for fixes (likely just a hopeful bisect?).
In any case, you definitely need this commit (was also marked for stable): 0c535e0 v3.3-rc7 dm io: fix discard support
Quoted 66 lines · Zdenek Kabelac
I've enabled more kernel debugging tools - and this came out when running test suite:
Unsure if this could be source of troubles?
kernel BUG at
arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMPDEBUG_PAGEALLOC
CPU 1Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tunbridge stp llc psmouse
serio_rawcryptomgr aead arc4crypto_algapisr_modcdrom iwl3945iwl_legacymac80211 cfg80211 crypto e1000e evdev loopdm_modautofs4Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 6464CTO/6464CTO
RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090) Stack: ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 Call Trace: [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] [<ffffffff81050177>] process_one_work+0x197/0x460 [<ffffffff81050116>] ? process_one_work+0x136/0x460 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] [<ffffffff81050986>] worker_thread+0x126/0x2d0 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 [<ffffffff81056066>] kthread+0x96/0xa0 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8133b610>] ? gs_change+0xb/0xb Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 0000 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba RIP [<
ffffffff81021f00>]__phys_addr+0x60/0x70
RSP <ffff880136e41aa0>---[ end trace 040dfd9d68fccabc ]---Zdenek
Dne 2.7.2012 15:33, Mike Snitzer napsal(a):
Quoted 7 lines
Not sure why you're testing such old kernels without regard for fixes (likely just a hopeful bisect?).
In any case, you definitely need this commit (was also marked for stable): 0c535e0 v3.3-rc7 dm io: fix discard support
After some more crashed filesystems here is the outcome with trial patch from Mikulas included:
https://bugzilla.redhat.com/show_bug.cgi?id=837607
Zdenek