Skip to contentSkip to conversation

Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel

linux-lvm Zdenek Kabelac 4 participants 14 messages Jun 30, 2012 – Jul 4, 2012

lore
Share & tools
lore
#0 lore raw

Hi

When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

My HW - t61, 4GB

Zdenek

BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
PGD 0
Oops: 0000 [#1] PREEMPT SMP
CPU 1
Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight

Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO

RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task ffff8801344ca310)
Stack:
 ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
Call Trace:
 [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
 [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
 [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
 [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
 [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
 [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
 [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
 [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
 [<ffffffff81069cf2>] kthread+0xa2/0xb0
 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
 [<ffffffff81533330>] ? gs_change+0xb/0xb
Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
 RSP <ffff880126b77c68>

CR2: 0000000000000178

# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug fs entries on failure of file creation and made extfrag_debug_root dentry local git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_LZMA=y
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_TREE_RCU_TRACE=y
CONFIG_IKCONFIG=m
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y

CONFIG_PARTITION_ADVANCED=y
CONFIG_AMIGA_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_LDM_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=m
CONFIG_DEFAULT_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_FREEZER=y

CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MEMTEST=y
CONFIG_MCORE2=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_NR_CPUS=2
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_EFI=y
CONFIG_HZ_300=y
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE=""
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
CONFIG_PM_WAKELOCKS_GC=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_CAN_PM_TRACE=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_CUSTOM_METHOD=m

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_P4_CLOCKMOD=m

CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_STUB=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_X86_X32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y

CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_MROUTE=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_TUNNEL=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_DEFAULT_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_IPV6=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m

CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m

CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TIME=m

CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_NONE=y
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_LLC=m
CONFIG_DNS_RESOLVER=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_BQL=y

CONFIG_NET_PKTGEN=m
CONFIG_IRDA=m

CONFIG_IRTTY_SIR=m

CONFIG_BT=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_HIDP=m

CONFIG_BT_HCIBTUSB=m
CONFIG_WIRELESS=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_CFG80211=m
CONFIG_CFG80211_DEVELOPER_WARNINGS=y
CONFIG_CFG80211_DEFAULT_PS=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_CFG80211_WEXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
CONFIG_LIB80211=m
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_RFKILL=y
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_RFKILL_REGULATOR=m
CONFIG_HAVE_BPF_JIT=y

CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_DMA_SHARED_BUFFER=y
CONFIG_CONNECTOR=m
CONFIG_MTD=m
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1

CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_SM_FTL=m

CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y

CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m

CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND_ECC_SMC=y

CONFIG_PNP=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_VIRTIO_BLK=y

CONFIG_INTEL_MEI=m
CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_VIRTIO=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

CONFIG_SATA_AHCI=y
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_ATA_PIIX=y

CONFIG_PATA_MPIIX=y

CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MULTICORE_RAID456=y
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m

CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_DUMMY=m
CONFIG_MII=m
CONFIG_NETCONSOLE=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m

CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E1000E=m
CONFIG_IXGBEVF=m
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_ASYNC=m
CONFIG_SLHC=m

CONFIG_WLAN=y
CONFIG_IWLWIFI=m

CONFIG_IWLEGACY=m
CONFIG_IWL3945=m

CONFIG_IWLEGACY_DEBUGFS=y

CONFIG_INPUT=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_EVDEV=m

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_SERIAL=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_UINPUT=m

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m

CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVKMEM=y

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=16
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_HVC_DRIVER=y
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m

CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m

CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m

CONFIG_I2C_TINY_USB=m

CONFIG_I2C_DEBUG_ALGO=y
CONFIG_HSI=m
CONFIG_HSI_BOARDINFO=y

CONFIG_HSI_CHAR=m

CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_POWER_SUPPLY=y
CONFIG_HWMON=y

CONFIG_SENSORS_ACPI_POWER=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y

CONFIG_SOFT_WATCHDOG=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y

CONFIG_SSB_POSSIBLE=y

CONFIG_BCMA_POSSIBLE=y

CONFIG_MFD_CORE=m
CONFIG_LPC_SCH=m
CONFIG_LPC_ICH=m
CONFIG_REGULATOR=y
CONFIG_REGULATOR_DEBUG=y
CONFIG_REGULATOR_DUMMY=y
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
CONFIG_MEDIA_SUPPORT=m

CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m

CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_RC_LOOPBACK=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_CUSTOMISE=y

CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_V4L2_MEM2MEM_DEV=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
CONFIG_VIDEO_ADV_DEBUG=y
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m

CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m

CONFIG_VIDEO_SAA6588=m

CONFIG_VIDEO_SAA711X=m

CONFIG_VIDEO_CX25840=m

CONFIG_VIDEO_CX2341X=m

CONFIG_VIDEO_VIVI=m
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
CONFIG_V4L_PCI_DRIVERS=y
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_TESTDEV=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y

CONFIG_DVB_USB=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_TTUSB_BUDGET=m

CONFIG_DVB_FE_CUSTOMISE=y

CONFIG_DVB_TDA8261=m
CONFIG_DVB_TDA10071=m

CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_AF9013=m

CONFIG_DVB_DIB8000=m

CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m

CONFIG_DVB_DUMMY_FE=m

CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m

CONFIG_DRM_I915=m
CONFIG_DRM_MGA=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_GENERIC=m

CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_DUMMY=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_PCI=y
CONFIG_SND_EMU10K1=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_PREALLOC_SIZE=64
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=500
CONFIG_SND_INTEL8X0=m
CONFIG_AC97_BUS=m

CONFIG_HID=m
CONFIG_HID_GENERIC=m

CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m

CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB_ARCH_HAS_XHCI=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=m
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=m
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

CONFIG_USB_SUSPEND=y
CONFIG_USB_MON=m

CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m

CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m

CONFIG_USB_STORAGE=m
CONFIG_USB_UAS=m

CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m

CONFIG_USB_TEST=m

CONFIG_MMC=m

CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y

CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

CONFIG_LEDS_TRIGGERS=y

CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m

CONFIG_EDAC=y

CONFIG_EDAC_DEBUG=y
CONFIG_EDAC_DECODE_MCE=y
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_E752X=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"

CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y

CONFIG_RTC_DRV_CMOS=y

CONFIG_DMADEVICES=y

CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y

CONFIG_NET_DMA=y
CONFIG_DCA=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y

CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=m
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y

CONFIG_STAGING=y
CONFIG_ZRAM=m
CONFIG_ZRAM_DEBUG=y
CONFIG_ZCACHE=y
CONFIG_ZSMALLOC=y

CONFIG_USB_WPAN_HCD=m
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
CONFIG_THINKPAD_ACPI_DEBUG=y
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_SENSORS_HDAPS=y
CONFIG_ACPI_WMI=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_INTEL_IPS=m

CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
CONFIG_IOMMU_SUPPORT=y

CONFIG_VIRT_DRIVERS=y
CONFIG_PM_DEVFREQ=y

CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
CONFIG_DEVFREQ_GOV_PERFORMANCE=y
CONFIG_DEVFREQ_GOV_POWERSAVE=y
CONFIG_DEVFREQ_GOV_USERSPACE=y

CONFIG_EDD=m
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=m
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=m

CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=m
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y

CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
CONFIG_FSCACHE_HISTOGRAM=y
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m
CONFIG_CACHEFILES_DEBUG=y

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=852
CONFIG_FAT_DEFAULT_IOCHARSET="cp1250"
CONFIG_NTFS_FS=m
CONFIG_NTFS_RW=y

CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m CONFIG_MISC_FILESYSTEMS=y
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_LZO=y
CONFIG_JFFS2_RTIME=y
CONFIG_JFFS2_RUBIN=y CONFIG_JFFS2_CMODE_PRIORITY=y
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_XZ=y CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_CIFS=m
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_852=m CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
CONFIG_SLUB_STATS=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RCU=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
CONFIG_RCU_TRACE=y
CONFIG_LKDTM=m
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_UPROBE_EVENT=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_MMIOTRACE=y
CONFIG_MMIOTRACE_TEST=m
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DMA_API_DEBUG=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_RODATA=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_OPTIMIZE_INLINING=y

CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
CONFIG_INTEGRITY=y
CONFIG_EVM=y
CONFIG_DEFAULT_SECURITY_SELINUX=y
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m

CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_XTS=m

CONFIG_CRYPTO_HMAC=y

CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=y
CONFIG_CRYPTO_SHA256=y

CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m

CONFIG_CRYPTO_LZO=y

CONFIG_CRYPTO_HW=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
CONFIG_BINARY_PRINTF=y

CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC32_SLICEBY8=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
CONFIG_DDR=y

#1 lore raw

Hi

When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

My HW - t61, 4GB

Zdenek

BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
PGD 0
Oops: 0000 [#1] PREEMPT SMP
CPU 1
Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight

Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO

RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task ffff8801344ca310)
Stack:
 ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
Call Trace:
 [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
 [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
 [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
 [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
 [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
 [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
 [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
 [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
 [<ffffffff81069cf2>] kthread+0xa2/0xb0
 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
 [<ffffffff81533330>] ? gs_change+0xb/0xb
Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
 RSP <ffff880126b77c68>

CR2: 0000000000000178

# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug fs entries on failure of file creation and made extfrag_debug_root dentry local git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_LZMA=y
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_TREE_RCU_TRACE=y
CONFIG_IKCONFIG=m
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y

CONFIG_PARTITION_ADVANCED=y
CONFIG_AMIGA_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_LDM_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=m
CONFIG_DEFAULT_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_FREEZER=y

CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MEMTEST=y
CONFIG_MCORE2=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_NR_CPUS=2
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_EFI=y
CONFIG_HZ_300=y
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE=""
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
CONFIG_PM_WAKELOCKS_GC=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_CAN_PM_TRACE=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_CUSTOM_METHOD=m

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_P4_CLOCKMOD=m

CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_STUB=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_X86_X32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y

CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_MROUTE=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_TUNNEL=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_DEFAULT_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_IPV6=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m

CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m

CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TIME=m

CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_NONE=y
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_LLC=m
CONFIG_DNS_RESOLVER=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_BQL=y

CONFIG_NET_PKTGEN=m
CONFIG_IRDA=m

CONFIG_IRTTY_SIR=m

CONFIG_BT=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_HIDP=m

CONFIG_BT_HCIBTUSB=m
CONFIG_WIRELESS=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_CFG80211=m
CONFIG_CFG80211_DEVELOPER_WARNINGS=y
CONFIG_CFG80211_DEFAULT_PS=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_CFG80211_WEXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
CONFIG_LIB80211=m
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_RFKILL=y
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_RFKILL_REGULATOR=m
CONFIG_HAVE_BPF_JIT=y

CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_DMA_SHARED_BUFFER=y
CONFIG_CONNECTOR=m
CONFIG_MTD=m
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1

CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_SM_FTL=m

CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y

CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m

CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND_ECC_SMC=y

CONFIG_PNP=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_VIRTIO_BLK=y

CONFIG_INTEL_MEI=m
CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_VIRTIO=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

CONFIG_SATA_AHCI=y
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_ATA_PIIX=y

CONFIG_PATA_MPIIX=y

CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MULTICORE_RAID456=y
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m

CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_DUMMY=m
CONFIG_MII=m
CONFIG_NETCONSOLE=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m

CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E1000E=m
CONFIG_IXGBEVF=m
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_ASYNC=m
CONFIG_SLHC=m

CONFIG_WLAN=y
CONFIG_IWLWIFI=m

CONFIG_IWLEGACY=m
CONFIG_IWL3945=m

CONFIG_IWLEGACY_DEBUGFS=y

CONFIG_INPUT=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_EVDEV=m

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_SERIAL=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_UINPUT=m

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m

CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVKMEM=y

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=16
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_HVC_DRIVER=y
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m

CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m

CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m

CONFIG_I2C_TINY_USB=m

CONFIG_I2C_DEBUG_ALGO=y
CONFIG_HSI=m
CONFIG_HSI_BOARDINFO=y

CONFIG_HSI_CHAR=m

CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_POWER_SUPPLY=y
CONFIG_HWMON=y

CONFIG_SENSORS_ACPI_POWER=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y

CONFIG_SOFT_WATCHDOG=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y

CONFIG_SSB_POSSIBLE=y

CONFIG_BCMA_POSSIBLE=y

CONFIG_MFD_CORE=m
CONFIG_LPC_SCH=m
CONFIG_LPC_ICH=m
CONFIG_REGULATOR=y
CONFIG_REGULATOR_DEBUG=y
CONFIG_REGULATOR_DUMMY=y
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
CONFIG_MEDIA_SUPPORT=m

CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m

CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_RC_LOOPBACK=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_CUSTOMISE=y

CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_V4L2_MEM2MEM_DEV=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
CONFIG_VIDEO_ADV_DEBUG=y
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m

CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m

CONFIG_VIDEO_SAA6588=m

CONFIG_VIDEO_SAA711X=m

CONFIG_VIDEO_CX25840=m

CONFIG_VIDEO_CX2341X=m

CONFIG_VIDEO_VIVI=m
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
CONFIG_V4L_PCI_DRIVERS=y
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_TESTDEV=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y

CONFIG_DVB_USB=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_TTUSB_BUDGET=m

CONFIG_DVB_FE_CUSTOMISE=y

CONFIG_DVB_TDA8261=m
CONFIG_DVB_TDA10071=m

CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_AF9013=m

CONFIG_DVB_DIB8000=m

CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m

CONFIG_DVB_DUMMY_FE=m

CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m

CONFIG_DRM_I915=m
CONFIG_DRM_MGA=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_GENERIC=m

CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_DUMMY=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_PCI=y
CONFIG_SND_EMU10K1=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_PREALLOC_SIZE=64
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=500
CONFIG_SND_INTEL8X0=m
CONFIG_AC97_BUS=m

CONFIG_HID=m
CONFIG_HID_GENERIC=m

CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m

CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB_ARCH_HAS_XHCI=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=m
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=m
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

CONFIG_USB_SUSPEND=y
CONFIG_USB_MON=m

CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m

CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m

CONFIG_USB_STORAGE=m
CONFIG_USB_UAS=m

CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m

CONFIG_USB_TEST=m

CONFIG_MMC=m

CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y

CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

CONFIG_LEDS_TRIGGERS=y

CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m

CONFIG_EDAC=y

CONFIG_EDAC_DEBUG=y
CONFIG_EDAC_DECODE_MCE=y
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_E752X=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"

CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y

CONFIG_RTC_DRV_CMOS=y

CONFIG_DMADEVICES=y

CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y

CONFIG_NET_DMA=y
CONFIG_DCA=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y

CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=m
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y

CONFIG_STAGING=y
CONFIG_ZRAM=m
CONFIG_ZRAM_DEBUG=y
CONFIG_ZCACHE=y
CONFIG_ZSMALLOC=y

CONFIG_USB_WPAN_HCD=m
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
CONFIG_THINKPAD_ACPI_DEBUG=y
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_SENSORS_HDAPS=y
CONFIG_ACPI_WMI=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_INTEL_IPS=m

CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
CONFIG_IOMMU_SUPPORT=y

CONFIG_VIRT_DRIVERS=y
CONFIG_PM_DEVFREQ=y

CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
CONFIG_DEVFREQ_GOV_PERFORMANCE=y
CONFIG_DEVFREQ_GOV_POWERSAVE=y
CONFIG_DEVFREQ_GOV_USERSPACE=y

CONFIG_EDD=m
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=m
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=m

CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=m
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y

CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
CONFIG_FSCACHE_HISTOGRAM=y
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m
CONFIG_CACHEFILES_DEBUG=y

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=852
CONFIG_FAT_DEFAULT_IOCHARSET="cp1250"
CONFIG_NTFS_FS=m
CONFIG_NTFS_RW=y

CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m CONFIG_MISC_FILESYSTEMS=y
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_LZO=y
CONFIG_JFFS2_RTIME=y
CONFIG_JFFS2_RUBIN=y CONFIG_JFFS2_CMODE_PRIORITY=y
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_XZ=y CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_CIFS=m
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_852=m CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
CONFIG_SLUB_STATS=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RCU=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
CONFIG_RCU_TRACE=y
CONFIG_LKDTM=m
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_UPROBE_EVENT=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_MMIOTRACE=y
CONFIG_MMIOTRACE_TEST=m
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DMA_API_DEBUG=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_RODATA=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_OPTIMIZE_INLINING=y

CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
CONFIG_INTEGRITY=y
CONFIG_EVM=y
CONFIG_DEFAULT_SECURITY_SELINUX=y
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m

CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_XTS=m

CONFIG_CRYPTO_HMAC=y

CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=y
CONFIG_CRYPTO_SHA256=y

CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m

CONFIG_CRYPTO_LZO=y

CONFIG_CRYPTO_HW=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
CONFIG_BINARY_PRINTF=y

CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC32_SLICEBY8=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
CONFIG_DDR=y

#2 lore raw
In reply to Zdenek Kabelac
Quoted 5 lines · Zdenek Kabelac

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 4 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?

I see no mention of madvise or MADV_REMOVE or fallocate or anything related in that git tree.

If you have something else running at the same time, which happens to use madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Any ideas from the DM guys? Has anyone else seen anything like this?

Do all your oopses look like this one?

Thanks,
Hugh

Quoted 112 lines

My HW - t61, 4GB

Zdenek

BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
PGD 0
Oops: 0000 [#1] PREEMPT SMP
CPU 1
Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq

async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight

Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO

RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>]
dm_rh_region_context+0xd/0x20 [dm_region_hash]
RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task
ffff8801344ca310)
Stack:
 ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
Call Trace:
 [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
 [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
 [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
 [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
 [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
 [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
 [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
 [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
 [<ffffffff81069cf2>] kthread+0xa2/0xb0
 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
 [<ffffffff81533330>] ? gs_change+0xb/0xb
Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8

c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
RSP <ffff880126b77c68>
CR2: 0000000000000178

# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of
git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug fs entries on failure of file creation and made extfrag_debug_root dentry local
git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

#3 lore raw

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 11 lines · Zdenek Kabelac
Quoted 5 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 9 lines

I see no mention of madvise or MADV_REMOVE or fallocate or anything related in that git tree.

If you have something else running at the same time, which happens to use madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB)

Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.

So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.

Quoted 3 lines

I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

I could try that if that will help.

Quoted 1 line

I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Shouldn't be hard to run - unsure if every config setup is influnenced or just mine config.

Quoted 4 lines

Any ideas from the DM guys? Has anyone else seen anything like this?

Do all your oopses look like this one?

I think I've get yet another one - but also within dm_rh_region

It could be that your patch exposed problem of some different part of stack - not really sure - it's just now with 3.5 this crash will not allow to pass whole test suite - I've tried also in kvm machine and it's been reproducible (so in the worst case I could eventually send you 2GB image)

The problem is - there is not a 'single test case' to trigger the oops (at least I've not figured out one) - it's the combination of multiple tests running after each other - but for simplication this should be enough:

make check_local T=shell/lvconvert

Which usually dies on shell/lvconvert-repair-transient.sh

Zdenek

#4 lore raw
Quoted 41 lines · Zdenek Kabelac

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 12 lines · Zdenek Kabelac
Quoted 6 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 9 lines

I see no mention of madvise or MADV_REMOVE or fallocate or anything related in that git tree.

If you have something else running at the same time, which happens to use madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

Quoted 6 lines

What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB)

Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Quoted 3 lines

So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

Quoted 6 lines
Quoted 4 lines

I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

I could try that if that will help.

That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before.

Quoted 5 lines
Quoted 2 lines

I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Shouldn't be hard to run - unsure if every config setup is influnenced or just mine config.

I'll start from your config.

Quoted 13 lines
Quoted 5 lines

Any ideas from the DM guys? Has anyone else seen anything like this?

Do all your oopses look like this one?

I think I've get yet another one - but also within dm_rh_region

It could be that your patch exposed problem of some different part of stack - not really sure - it's just now with 3.5 this crash will not allow to pass whole test suite - I've tried also in kvm machine and it's been reproducible (so in the worst case I could eventually send you 2GB image)

Quoted 7 lines

The problem is - there is not a 'single test case' to trigger the oops (at least I've not figured out one) - it's the combination of multiple tests running after each other - but for simplication this should be enough:

make check_local T=shell/lvconvert

Which usually dies on shell/lvconvert-repair-transient.sh

Thanks again, I'll report back later.

Hugh

#5 lore raw

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):

Quoted 66 lines · Zdenek Kabelac
Quoted 37 lines

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 12 lines · Zdenek Kabelac
Quoted 6 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 5 lines

If you have something else running at the same time, which happens to use madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

Quoted 6 lines

What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB)

Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Quoted 3 lines

So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Quoted 3 lines

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)

So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

And you are correct, the first commit which causes crash really is
83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'make check_local' execution - and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.

So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

I could try that if that will help.

That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before.

the only thing which could be tricky is udev support
(by default it's not enabled ---enable-udev_sync)
However Debian based distros are distributing their own rules, which are not 100% compatible with upstream and create some unpredictable issues, where the slowness is the least problem.)

If you have Fedora Rawhide with latest lvm2 installed - you should get pretty well configured system for running test dir (unfortunately there is no way to virtualize udev...)

Zdenek

#6 lore raw
Quoted 49 lines · Zdenek Kabelac

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):

Quoted 48 lines · Zdenek Kabelac
Quoted 47 lines

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 16 lines · Zdenek Kabelac
Quoted 8 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

Quoted 7 lines
Quoted 1 line

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 9 lines

If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

and using loop device to simulate device (small size - it should fit bellow
200MB)

Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

Quoted 26 lines
Quoted 6 lines
Quoted 4 lines

So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

From your later remarks, I inferred tmpfs.

Quoted 13 lines

If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Quoted 3 lines

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

Quoted 8 lines

While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)

So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

Quoted 10 lines

And you are correct, the first commit which causes crash really is 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'make check_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in the first pass.

Quoted 5 lines

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.

So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.

Quoted 20 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

Quoted 5 lines
Quoted 1 line

I could try that if that will help.

That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before.

the only thing which could be tricky is udev support
(by default it's not enabled ---enable-udev_sync)
However Debian based distros are distributing their own rules, which are not 100% compatible with upstream and create some unpredictable issues, where the slowness is the least problem.)

Quoted 3 lines

If you have Fedora Rawhide with latest lvm2 installed - you should get pretty
well configured system for running test dir (unfortunately there is no way
to virtualize udev...)

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io().

But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.

I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.)

I've added a few people to the Cc, innocents fingered by git blame on some REQ_DISCARD lines in drivers/md. But I'll back out myself:
I'm not short of other things to look at, and have little to offer in drivers/md.

Hugh

#7 lore raw

Dne 1.7.2012 20:45, Hugh Dickins napsal(a):

Quoted 121 lines · Zdenek Kabelac
Quoted 49 lines

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):

Quoted 48 lines · Zdenek Kabelac
Quoted 47 lines

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 16 lines · Zdenek Kabelac
Quoted 8 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

Quoted 7 lines
Quoted 1 line

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 9 lines

If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

and using loop device to simulate device (small size - it should fit bellow
200MB)

Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

From your later remarks, I inferred tmpfs.

Quoted 17 lines
Quoted 13 lines

If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Quoted 3 lines

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

Quoted 8 lines

While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)

So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

Quoted 13 lines

And you are correct, the first commit which causes crash really is 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'make check_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.

Quoted 6 lines
Quoted 2 lines

So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.

Well I've played meanwhile the game with minimized kernel config and the outcome is:

last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef

dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices

Quoted 3 lines

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io().

Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.

Quoted 9 lines

But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.

I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.)

So does anyone has some idea what should be checked next ?

Zdenek

#8 lore raw

Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):

Quoted 149 lines

Dne 1.7.2012 20:45, Hugh Dickins napsal(a):

Quoted 121 lines · Zdenek Kabelac
Quoted 49 lines

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):

Quoted 48 lines · Zdenek Kabelac
Quoted 47 lines

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 16 lines · Zdenek Kabelac
Quoted 8 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

Quoted 7 lines
Quoted 1 line

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 9 lines

If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

and using loop device to simulate device (small size - it should fit bellow
200MB)

Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

From your later remarks, I inferred tmpfs.

Quoted 17 lines
Quoted 13 lines

If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Quoted 3 lines

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

Quoted 8 lines

While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :)

So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

Quoted 13 lines

And you are correct, the first commit which causes crash really is 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'make check_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config.

Quoted 6 lines
Quoted 2 lines

So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.

Well I've played meanwhile the game with minimized kernel config and the outcome is:

last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef

dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices

Quoted 3 lines

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io().

Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.

Quoted 4 lines

But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.

Quoted 7 lines
Quoted 5 lines

I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.)

So does anyone has some idea what should be checked next ?

I've enabled more kernel debugging tools - and this came out when running test suite:

Unsure if this could be source of troubles?

kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1

Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4

Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 6464CTO/6464CTO

RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090)
Stack:
 ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
 [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
 [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
 [<ffffffff81050177>] process_one_work+0x197/0x460
 [<ffffffff81050116>] ? process_one_work+0x136/0x460
 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
 [<ffffffff81050986>] worker_thread+0x126/0x2d0
 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
 [<ffffffff81056066>] kthread+0x96/0xa0
 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8133b610>] ? gs_change+0xb/0xb
Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
 RSP <ffff880136e41aa0>
---[ end trace 040dfd9d68fccabc ]---

Zdenek

#9 lore raw
Quoted 10 lines · Zdenek Kabelac

Date: Mon, 02 Jul 2012 00:03:07 +0200
From: Zdenek Kabelac <[email protected]>
To: Hugh Dickins <[email protected]>
Cc: Mike Snitzer <[email protected]>, Mikulas Patocka <[email protected]>,
Joe Thornber <[email protected]>,
LVM general discussion and development <[email protected]>, [email protected], Alasdair G Kergon <[email protected]>, [email protected], Lukas Czerner <[email protected]>
Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel

Quoted 174 lines

Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):

Quoted 173 lines

Dne 1.7.2012 20:45, Hugh Dickins napsal(a):

Quoted 139 lines · Zdenek Kabelac
Quoted 54 lines

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):

Quoted 53 lines · Zdenek Kabelac
Quoted 52 lines

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 18 lines · Zdenek Kabelac
Quoted 10 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit
this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does
not
included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

Quoted 7 lines
Quoted 1 line

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 11 lines

If you have something else running at the same time, which happens to
use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in LVM_TEST_DIR (default is
/tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards.

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

and using loop device to simulate device (small size - it should fit bellow
200MB)

Within this file second layer through virtual DM devices is created and
simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

So since everything now support TRIM - such operations should be passed
down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

From your later remarks, I inferred tmpfs.

Quoted 25 lines
Quoted 15 lines

If tmpfs, then it will make much more sense if we assume your bisection
endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might
have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Quoted 3 lines

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

While I've been aware of the fact that tmpfs was enhanced with trim support -
I've not tried to run on real ext4 filesystem since for my tests I'm using
tmpfs for quite some time to safe rewrites of SSD :)

Quoted 4 lines

So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

Quoted 20 lines

And you are correct, the first commit which causes crash really is 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've
missed to properly identify this commit in my bisect is that crash usually
happens on the second pass of the lvm test suite 'make check_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on
3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in
the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
and it's crashing as well - so it's probably not uniqueness of my config.

Quoted 7 lines
Quoted 3 lines

So is there any primary suspect in 3.2 which is worth to check - or I need
another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.

Well I've played meanwhile the game with minimized kernel config and the outcome is:

last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef

dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices

Quoted 3 lines

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io().

Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.

Quoted 9 lines

But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.

I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.)

Hi Zdenek,

Quoted 2 lines
Quoted 2 lines

So does anyone has some idea what should be checked next ?

I've missed the information in the thread, but since you're the most familiar with the lvm test suite, could you distil the reproducer which does not require to run lvm test suit ?

Quoted 7 lines

I've enabled more kernel debugging tools - and this came out when running test suite:

Unsure if this could be source of troubles?

Quoted 8 lines

kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1

Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp

llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4

Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO

So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ?

Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ?

Looking at git log, this commit has been brought to my attention:

0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support

seems related to this crash.

Please retest with recent kernel.

Thanks!
-Lukas

Quoted 53 lines

6464CTO/6464CTO

RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task
ffff880136e3e090)
Stack:
 ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
 [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
 [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
 [<ffffffff81050177>] process_one_work+0x197/0x460
 [<ffffffff81050116>] ? process_one_work+0x136/0x460
 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
 [<ffffffff81050986>] worker_thread+0x126/0x2d0
 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
 [<ffffffff81056066>] kthread+0x96/0xa0
 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8133b610>] ? gs_change+0xb/0xb
Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00

78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP <ffff880136e41aa0>

---[ end trace 040dfd9d68fccabc ]---

Zdenek

#10 lore raw
Quoted 235 lines · Lukᅵ Czerner

Date: Mon, 2 Jul 2012 11:00:06 +0200 (CEST)
From: Lukᅵ Czerner <[email protected]>
To: Zdenek Kabelac <[email protected]>
Cc: Hugh Dickins <[email protected]>, Mike Snitzer <[email protected]>,
Mikulas Patocka <[email protected]>, Joe Thornber <[email protected]>, LVM general discussion and development <[email protected]>, [email protected], Alasdair G Kergon <[email protected]>, [email protected], Lukas Czerner <[email protected]>
Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel

Quoted 10 lines · Zdenek Kabelac

Date: Mon, 02 Jul 2012 00:03:07 +0200
From: Zdenek Kabelac <[email protected]>
To: Hugh Dickins <[email protected]>
Cc: Mike Snitzer <[email protected]>, Mikulas Patocka <[email protected]>,
Joe Thornber <[email protected]>,
LVM general discussion and development <[email protected]>, [email protected], Alasdair G Kergon <[email protected]>, [email protected], Lukas Czerner <[email protected]>
Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel

Quoted 174 lines

Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):

Quoted 173 lines

Dne 1.7.2012 20:45, Hugh Dickins napsal(a):

Quoted 139 lines · Zdenek Kabelac
Quoted 54 lines

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):

Quoted 53 lines · Zdenek Kabelac
Quoted 52 lines

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):

Quoted 18 lines · Zdenek Kabelac
Quoted 10 lines

When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit
this
oops
is to run lvm test suite for 10 minutes. Since 3.5 merge window does
not
included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on.

Quoted 3 lines

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

Quoted 7 lines
Quoted 1 line

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at
git.fedorahosted.org/git/lvm2.git
under tests/ ?

Yes - that's it -

make
as root:
cd test
make check_local

(inside test subdirectory should be enough, if not - just report any problem)

Quoted 11 lines

If you have something else running at the same time, which happens to
use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in LVM_TEST_DIR (default is
/tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards.

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

and using loop device to simulate device (small size - it should fit bellow
200MB)

Within this file second layer through virtual DM devices is created and
simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

So since everything now support TRIM - such operations should be passed
down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

From your later remarks, I inferred tmpfs.

Quoted 25 lines
Quoted 15 lines

If tmpfs, then it will make much more sense if we assume your bisection
endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might
have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing.

Quoted 3 lines

Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant.

While I've been aware of the fact that tmpfs was enhanced with trim support -
I've not tried to run on real ext4 filesystem since for my tests I'm using
tmpfs for quite some time to safe rewrites of SSD :)

Quoted 4 lines

So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

Quoted 20 lines

And you are correct, the first commit which causes crash really is 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've
missed to properly identify this commit in my bisect is that crash usually
happens on the second pass of the lvm test suite 'make check_local' execution
- and I've been running test just once. To be sure I've run 5 run passes on
3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in
the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
and it's crashing as well - so it's probably not uniqueness of my config.

Quoted 7 lines
Quoted 3 lines

So is there any primary suspect in 3.2 which is worth to check - or I need
another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then.

Well I've played meanwhile the game with minimized kernel config and the outcome is:

last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef

dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices

Quoted 3 lines

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io().

Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off.

Quoted 9 lines

But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM.

I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.)

Hi Zdenek,

Quoted 2 lines
Quoted 2 lines

So does anyone has some idea what should be checked next ?

I've missed the information in the thread, but since you're the most familiar with the lvm test suite, could you distil the reproducer which does not require to run lvm test suit ?

Quoted 7 lines

I've enabled more kernel debugging tools - and this came out when running test suite:

Unsure if this could be source of troubles?

Quoted 8 lines

kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1

Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp

llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4

Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO

So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ?

Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ?

Looking at git log, this commit has been brought to my attention:

Quoted 61 lines

0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support

seems related to this crash.

Please retest with recent kernel.

Thanks!
-Lukas

Quoted 52 lines

6464CTO/6464CTO

RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task
ffff880136e3e090)
Stack:
 ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
 [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
 [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
 [<ffffffff81050177>] process_one_work+0x197/0x460
 [<ffffffff81050116>] ? process_one_work+0x136/0x460
 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
 [<ffffffff81050986>] worker_thread+0x126/0x2d0
 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
 [<ffffffff81056066>] kthread+0x96/0xa0
 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8133b610>] ? gs_change+0xb/0xb
Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00

78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP <ffff880136e41aa0>

---[ end trace 040dfd9d68fccabc ]---

Zdenek

So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
(https://lkml.org/lkml/2012/6/30/98) I think that this is problem in the device mapper itself. I do not think it has anything to do with tmpfs or mm. According to bisects from Zdenek it clearly shows that the problem appear when the discard support for the loop device is added, so it is most likely related to the dm discard support.

Anyway, the backtrace points to the NULL pointed dereference in dm_rh_region_context() which is simple function:

void *dm_rh_region_context(struct dm_region *reg)

{
       return reg->rh->context;
}

so either reg, or reg-rh is NULL. Now the only place this is used is from recovery_complete() in dm-raid1.c. So this is somewhat related to raid recovery. I am not familiar with the dm code, but can someone from the dm team look at this ?

But just to be sure to rule out the punch hole thing Zdenek can you run your tests on the "real" discard capable device ? Or at least on the device which does not convert discard requests into punch hole ? You can use scsi_debug to create such device:

modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1

Thanks!
-Lukas

#12 lore raw
In reply to Zdenek Kabelac

Not sure why you're testing such old kernels without regard for fixes (likely just a hopeful bisect?).

In any case, you definitely need this commit (was also marked for stable): 0c535e0 v3.3-rc7 dm io: fix discard support

Quoted 66 lines · Zdenek Kabelac

I've enabled more kernel debugging tools - and this came out when running test suite:

Unsure if this could be source of troubles?

kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1

Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun

bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4

Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 6464CTO/6464CTO

RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090)
Stack:
 ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
 [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
 [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
 [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
 [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
 [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
 [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
 [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
 [<ffffffff81050177>] process_one_work+0x197/0x460
 [<ffffffff81050116>] ? process_one_work+0x136/0x460
 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
 [<ffffffff81050986>] worker_thread+0x126/0x2d0
 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
 [<ffffffff81056066>] kthread+0x96/0xa0
 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
 [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8133b610>] ? gs_change+0xb/0xb
Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00

00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP <ffff880136e41aa0>

---[ end trace 040dfd9d68fccabc ]---

Zdenek

#11 lore raw
In reply to Lukáš Czerner
Quoted 11 lines · Lukáš Czerner
Quoted 11 lines

So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ?

Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ?

Quoted 7 lines
Quoted 7 lines

Looking at git log, this commit has been brought to my attention:

0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support

seems related to this crash.

Please retest with recent kernel.

Ah, you beat me to recommending that fix ;)

Quoted 6 lines

So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
(https://lkml.org/lkml/2012/6/30/98) I think that this is problem in the device mapper itself. I do not think it has anything to do with tmpfs or mm. According to bisects from Zdenek it clearly shows that the problem appear when the discard support for the loop device is added, so it is most likely related to the dm discard support.

What about using scsi_debug with the dm-mirror target?

Never say never, DM-mirror and/or dm-io code could still have an issue, but the commit referenced above did fix discard with the mirror target back in 3.3.

Quoted 4 lines

Anyway, the backtrace points to the NULL pointed dereference in dm_rh_region_context() which is simple function:

void *dm_rh_region_context(struct dm_region *reg)

Quoted 8 lines
{
       return reg->rh->context;
}

so either reg, or reg-rh is NULL. Now the only place this is used is from recovery_complete() in dm-raid1.c. So this is somewhat related to raid recovery. I am not familiar with the dm code, but can someone from the dm team look at this ?

I'll coordiinate with Zdenek.

Quoted 6 lines

But just to be sure to rule out the punch hole thing Zdenek can you run your tests on the "real" discard capable device ? Or at least on the device which does not convert discard requests into punch hole ? You can use scsi_debug to create such device:

modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1

Great minds think alike ;)

#13 lore raw
In reply to Mike Snitzer

Dne 2.7.2012 15:33, Mike Snitzer napsal(a):

Quoted 7 lines

Not sure why you're testing such old kernels without regard for fixes (likely just a hopeful bisect?).

In any case, you definitely need this commit (was also marked for stable): 0c535e0 v3.3-rc7 dm io: fix discard support

After some more crashed filesystems here is the outcome with trial patch from Mikulas included:

https://bugzilla.redhat.com/show_bug.cgi?id=837607

Zdenek