2.6.32.5 regression: page allocation failure. order:1,
Subsystems (2)
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
Downgrading now.
Logs available if you want them: http://rtr.ca/2.6.32_page_alloc_fail_logs.txt
--
Mark Lord
Mel Gorman
Acked-byQuoted 7 lines · Mark Lord
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
There was one bug related to MIGRATE_RESERVE that might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.
Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
==== CUT HERE ====
From: Hugh Dickins <[email protected]>
Subject: Fix 2.6.32 slowdown in heavy swapping
There is a problem with simply building kernels as part of a tmpfs loop swapping tests, and it's only obvious on the PowerPC G5. The problem is that those swapping builds run about 20% slower in 2.6.32 than 2.6.31 (and look as if they run increasingly slowly, though I'm not certain of that); and surprisingly it bisected down to your commit 5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 page-allocator: split per-cpu list into one-list-per-migrate-type
The problem was down to MIGRATE_RESERVE pages are being put on the MIGRATE_MOVABLE list, then freed as MIGRATE_MOVABLE. While it is not clear why this has such a severe impact, it may be down to how many short-lived high-order allocations are taking place. On machines making large numbers of short-lived-high-order allocations, they may be depending on MIGRATE_RESERVE to allocate in a timely fashion. In the case where they are GFP_ATOMIC, they may be depending on MIGRATE_RESERVE to just work.
The simplest, straight bugfix, patch is the one below: rely on page_private instead of migratetype when freeing.
--- mm/page_alloc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/page_alloc.c +3 -2
@@ -555,8 +555,9 @@ static void free_pcppages_bulk(struct zo page = list_entry(list->prev, struct page, lru);/* must delete as __free_one_page list manipulates */ list_del (&page->lru); -__free_one_page (page, zone,0 , migratetype); -trace_mm_page_pcpu_drain (page,0 , migratetype); +/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */ +__free_one_page (page, zone,0 ,page_private (page)); +trace_mm_page_pcpu_drain (page,0 ,page_private (page)); }while (--count && --batch_free && !list_empty (list)); }spin_unlock (&zone->lock);
Quoted 9 lines · Mel Gorman
Quoted 7 lines · Mark Lord
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
Quoted 8 lines
There was one bug related to
MIGRATE_RESERVEthat might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
..
We don't like to reboot our 24/7 server very often, and certainly not for debugging buggy kernels.
It's rock solid again with 2.6.31.12 on it now.
The defining characteristic of that machine, is that it has only 512MB of physical RAM. So perhaps I'll try booting a different machine here with mem=512M and see how that behaves. If the problem shows up on that, then I'll try the patch.
Thanks.
Quoted 14 lines
mm/page_alloc.c +3
-2@@ -555,8 +555,9 @@ static void free_pcppages_bulk(struct zo page = list_entry(list->prev, struct page, lru);/* must delete as __free_one_page list manipulates */ list_del (&page->lru); -__free_one_page (page, zone,0 , migratetype); -trace_mm_page_pcpu_drain (page,0 , migratetype); +/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */ +__free_one_page (page, zone,0 ,page_private (page)); +trace_mm_page_pcpu_drain (page,0 ,page_private (page)); }while (--count && --batch_free && !list_empty (list)); }spin_unlock (&zone->lock);
Quoted 6 lines · Mark Lord
Quoted 6 lines
There was one bug related to
MIGRATE_RESERVEthat might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.
Quoted 9 lines
Quoted 2 lines
Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
..
We don't like to reboot our 24/7 server very often, and certainly not for debugging buggy kernels.
It's rock solid again with 2.6.31.12 on it now.
Is there something specific about the workload that makes it easily reproducible? Are you saying that 2.6.31.12 is "rock solid" because it has survived a certain workload that caused these page allocation failures with 2.6.32.5, or simply because it has a longer uptime and hasn't shown a problem? It would be very helpful to describe the load so that we can attempt to reproduce it locally without a sacrifice to your server.
Quoted 5 lines · David Rientjes
Quoted 2 lines · Mark Lord
It's rock solid again with 2.6.31.12 on it now.
Is there something specific about the workload that makes it easily reproducible? Are you saying that 2.6.31.12 is "rock solid" because it has survived a certain workload that caused these page allocation failures with 2.6.32.5, or simply because it has a longer uptime and hasn't shown a problem? It would be very helpful to describe the load so that we can attempt to reproduce it locally without a sacrifice to your server.
..
Good questions, and I do "feel for you" here. :)
That machine has a light workload. Web server, email server, dhcp for the small local network, name server, etc.
It sits at one end of a 6mb/1.5mb DSL connection, hardly busy at all.
But the logfile posted shows many of those "allocation failures" within the first (only)36 hours of running 2.6.32.5:
mirrordir (doing a backup to a USB drive)
apcupsd (UPS monitoring; hardly an intensive activity) apache2 (the web server receives rather light use, and no fancy php or anything) apache2 (again!) apache2 (and again) apache2 (another) apache2 (yet again) ... apache2 (and again again) vim (hey, can't a guy edit his driver sources ??) cc1 (or compile them?)
So, not much happening, really.
It's a slow machine, a 600Mhz C7 ("VIA Samuel 2"), with only 512MB of RAM.
We've got 2.6.32.5 running on several other machines here with nary a glitch, but all of those have 2-4GB of RAM.
Cheers
Quoted 23 lines · Mark Lord
Quoted 9 lines · Mel Gorman
Quoted 7 lines · Mark Lord
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
Quoted 8 lines
There was one bug related to
MIGRATE_RESERVEthat might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
..
We don't like to reboot our 24/7 server very often, and certainly not for debugging buggy kernels.
Quoted 6 lines
It's rock solid again with 2.6.31.12 on it now.
The defining characteristic of that machine, is that it has only 512MB of physical RAM. So perhaps I'll try booting a different machine here with mem=512M and see how that behaves. If the problem shows up on that, then I'll try the patch.
..
Sod it. 2.6.32 is simply too broken for us here on 32-bit non-SMP.
Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook locks up at boot time with several repeated messages like this:
request_module: runaway loop modprobe binfmt_464c
Useless kernel on 32-bit. I hope 2.6.33 ends up less buggy.
Cheers
Quoted 32 lines · Mark Lord
Quoted 29 lines · Mark Lord
Quoted 9 lines · Mel Gorman
Quoted 7 lines · Mark Lord
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
Quoted 8 lines
There was one bug related to
MIGRATE_RESERVEthat might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
..
We don't like to reboot our 24/7 server very often, and certainly not for debugging buggy kernels.
It's rock solid again with 2.6.31.12 on it now.
The defining characteristic of that machine, is that it has only 512MB of physical RAM. So perhaps I'll try booting a different machine here with mem=512M and see how that behaves. If the problem shows up on that, then I'll try the patch.
..
Quoted 8 lines
Sod it. 2.6.32 is simply too broken for us here on 32-bit non-SMP.
Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook locks up at boot time with several repeated messages like this:
request_module: runaway loop modprobebinfmt_464cUseless kernel on 32-bit. I hope 2.6.33 ends up less buggy.
..
I rebuilt it (again!), this time as a pure UP (non-SMP) kernel, and it still locks at boot, with or without the mem=512M parameter.
This is one really bad kernel release for 32-bit x86.
-ml
Quoted 40 lines · Mark Lord
Quoted 39 lines · Mark Lord
Quoted 23 lines · Mark Lord
Quoted 9 lines · Mel Gorman
Quoted 7 lines · Mark Lord
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
Quoted 8 lines
There was one bug related to
MIGRATE_RESERVEthat might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
..
We don't like to reboot our 24/7 server very often, and certainly not for debugging buggy kernels.
Quoted 6 lines
It's rock solid again with 2.6.31.12 on it now.
The defining characteristic of that machine, is that it has only 512MB of physical RAM. So perhaps I'll try booting a different machine here with mem=512M and see how that behaves. If the problem shows up on that, then I'll try the patch.
..
Sod it. 2.6.32 is simply too broken for us here on 32-bit non-SMP.
Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook locks up at boot time with several repeated messages like this:
request_module: runaway loop modprobebinfmt_464c
Quoted 7 lines
Quoted 1 line
Useless kernel on 32-bit. I hope 2.6.33 ends up less buggy.
..
I rebuilt it (again!), this time as a pure UP (non-SMP) kernel, and it still locks at boot, with or without the mem=512M parameter.
This is one really bad kernel release for 32-bit x86.
FWFW, it doesn't seem to be anything generic.
My P4 box boots and runs fine nosmp+mem=512M.. no allocation errors so far while beating it up. It's not even a tiny bit fun to use with KDE while doing so though. Doing a bunch of git and nfs activity convinces reclaim that vital bits of my GUI are expendable.
-Mike
Quoted 40 lines · Mark Lord
Quoted 39 lines · Mark Lord
Quoted 23 lines · Mark Lord
Quoted 9 lines · Mel Gorman
Quoted 7 lines · Mark Lord
I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
Now, suddenly the logs are full of "page allocation failure. order:1", and the odd "page allocation failure. order:4" failures.
Wow. WTF happened in 2.6.32 ???
Quoted 8 lines
There was one bug related to
MIGRATE_RESERVEthat might be affecting you. It reported as impacting swap-orientated workloads but it could easily affect drivers that depend on high-order atomic allocations. Unfortunately, the fix is not signed-off yet but I expect it to make its way towards mainline when it is.Here is the patch with a slightly-altered changelog. Can you test if it makes a difference please?
..
We don't like to reboot our 24/7 server very often, and certainly not for debugging buggy kernels.
Quoted 6 lines
It's rock solid again with 2.6.31.12 on it now.
The defining characteristic of that machine, is that it has only 512MB of physical RAM. So perhaps I'll try booting a different machine here with mem=512M and see how that behaves. If the problem shows up on that, then I'll try the patch.
..
Sod it. 2.6.32 is simply too broken for us here on 32-bit non-SMP.
Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook locks up at boot time with several repeated messages like this:
request_module: runaway loop modprobebinfmt_464c
Quoted 7 lines
Quoted 1 line
Useless kernel on 32-bit. I hope 2.6.33 ends up less buggy.
..
I rebuilt it (again!), this time as a pure UP (non-SMP) kernel, and it still locks at boot, with or without the mem=512M parameter.
This is one really bad kernel release for 32-bit x86.
Have you attached a (2.6.31.x) dmesg output from the failing box somewhere or have I just missed it?
Rafael
Quoted 2 lines · Rafael J. Wysocki
Have you attached a (2.6.31.x) dmesg output from the failing box somewhere or have I just missed it?
..
The log from failing on 2.6.32.5 was linked to originally:
http://rtr.ca/2.6.32_page_alloc_fail_logs.txt
Here's the subsequent revert reboot back to 2.6.32.12:
http://rtr.ca/2.6.31.12_ok_logs.txt
In the time available thus far, I haven't managed to get 2.6.32.5 to boot/run on anything else, using either a UP kernel, or "nosmp", along with "mem=512M".
With a bit of luck, I'll have another similar VIA C7 box delivered here by tonight, and should be able to boot/run that off a cloned disk image from the original system.
Then we can poke at things a bit more.
If the box works, it will be 100% at our disposal.
Note that it's far too slow for practical direct use of git-bisect, taking hours to rebuild its own kernels. It might be possible, with a fair bit of effort, to arrange a quad-core box to do the kernel compiles for it, though.
Cheers
Quoted 19 lines · Mark Lord
Quoted 2 lines · Rafael J. Wysocki
Have you attached a (2.6.31.x) dmesg output from the failing box somewhere or have I just missed it?
..
The log from failing on 2.6.32.5 was linked to originally:
http://rtr.ca/2.6.32_page_alloc_fail_logs.txt
Here's the subsequent revert reboot back to 2.6.32.12:http://rtr.ca/2.6.31.12_ok_logs.txt
In the time available thus far, I haven't managed to get 2.6.32.5 to boot/run on anything else, using either a UP kernel, or "nosmp", along with "mem=512M".
With a bit of luck, I'll have another similar VIA C7 box delivered here by tonight, and should be able to boot/run that off a cloned disk image from the original system.
Quoted 7 lines
Then we can poke at things a bit more.
If the box works, it will be 100% at our disposal.Note that it's far too slow for practical direct use of git-bisect, taking hours to rebuild its own kernels. It might be possible, with a fair bit of effort, to arrange a quad-core box to do the kernel compiles for it, though.
..
Rather than wasting time trying to bisect a full major kernel revision, I think instead I'll just focus on mm/page_alloc.c.
It appears the biggest changes were to get_page_from_freelist(), so perhaps we'll just revert that one function and see how the clone system behaves.
In the meanwhile, here's the diff -u of the two .config files:
linux/.config +0 -0
@@ -1,7 +1,7 @@
#
# Automatically generated make config: don't edit
-# Linux kernel version: 2.6.31.12
-# Tue Jan 26 21:38:36 2010
+# Linux kernel version: 2.6.32.6
+# Tue Jan 26 20:57:44 2010
#
# CONFIG_64BIT is not setCONFIG_X86_32=y
@@ -16,7 +16,6 @@CONFIG_LOCKDEP_SUPPORT=yCONFIG_STACKTRACE_SUPPORT=yCONFIG_HAVE_LATENCYTOP_SUPPORT=y
-CONFIG_FAST_CMPXCHG_LOCAL=yCONFIG_MMU=yCONFIG_ZONE_DMA=yCONFIG_GENERIC_ISA_DMA=y
@@ -33,7 +32,8 @@CONFIG_ARCH_HAS_DEFAULT_IDLE=yCONFIG_ARCH_HAS_CACHE_LINE_SIZE=yCONFIG_HAVE_SETUP_PER_CPU_AREA=y
-CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y
+CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
+CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not setCONFIG_ARCH_HIBERNATION_POSSIBLE=yCONFIG_ARCH_SUSPEND_POSSIBLE=y
@@ -76,11 +76,12 @@
#
# RCU Subsystem
#
-CONFIG_CLASSIC_RCU=y
-# CONFIG_TREE_RCU is not set
-# CONFIG_PREEMPT_RCU is not set
+CONFIG_TREE_RCU=y
+# CONFIG_TREE_PREEMPT_RCU is not set
+# CONFIG_RCU_TRACE is not set
+CONFIG_RCU_FANOUT=8
+# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
-# CONFIG_PREEMPT_RCU_TRACE is not setCONFIG_IKCONFIG=yCONFIG_IKCONFIG_PROC=yCONFIG_LOG_BUF_SHIFT=15
@@ -123,21 +124,20 @@CONFIG_EVENTFD=yCONFIG_SHMEM=yCONFIG_AIO=y
-CONFIG_HAVE_PERF_COUNTERS=y
+CONFIG_HAVE_PERF_EVENTS=y
#
-# Performance Counters
+# Kernel Performance Events And Counters
#
+# CONFIG_PERF_EVENTS is not set
# CONFIG_PERF_COUNTERS is not setCONFIG_VM_EVENT_COUNTERS=yCONFIG_PCI_QUIRKS=y
-CONFIG_STRIP_ASM_SYMS=yCONFIG_COMPAT_BRK=yCONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
-# CONFIG_MARKERS is not setCONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not setCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
@@ -151,7 +151,7 @@
#
# GCOV-based kernel profiling
#
-# CONFIG_SLOW_WORK is not set
+CONFIG_SLOW_WORK=yCONFIG_HAVE_GENERIC_DMA_COHERENT=yCONFIG_SLABINFO=yCONFIG_RT_MUTEXES=y
@@ -217,6 +217,7 @@
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
+# CONFIG_MATOM is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not setCONFIG_X86_CPU=y
@@ -250,7 +251,6 @@
# CONFIG_PREEMPT is not set
# CONFIG_X86_UP_APIC is not set
# CONFIG_X86_MCE is not set
-# CONFIG_X86_ANCIENT_MCE is not setCONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
@@ -283,6 +283,7 @@CONFIG_VIRT_TO_BUS=yCONFIG_HAVE_MLOCK=yCONFIG_HAVE_MLOCKED_PAGE_BIT=y
+# CONFIG_KSM is not setCONFIG_DEFAULT_MMAP_MIN_ADDR=4096
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
# CONFIG_X86_RESERVE_LOW_64K is not set
@@ -313,6 +314,7 @@
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
+# CONFIG_PM_RUNTIME is not setCONFIG_ACPI=yCONFIG_ACPI_PROCFS=yCONFIG_ACPI_PROCFS_POWER=y
@@ -324,6 +326,7 @@CONFIG_ACPI_FAN=mCONFIG_ACPI_DOCK=yCONFIG_ACPI_PROCESSOR=m
+# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not setCONFIG_ACPI_THERMAL=m
# CONFIG_ACPI_CUSTOM_DSDT is not setCONFIG_ACPI_BLACKLIST_YEAR=0
@@ -332,6 +335,7 @@CONFIG_X86_PM_TIMER=yCONFIG_ACPI_CONTAINER=m
# CONFIG_ACPI_SBS is not set
+# CONFIG_SFI is not set
#
# CPU Frequency scaling
@@ -626,6 +630,7 @@
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_IP_SCTP is not set
+# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not setCONFIG_STP=m
@@ -723,6 +728,7 @@
# Generic Driver Options
#CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
+# CONFIG_DEVTMPFS is not setCONFIG_STANDALONE=yCONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_FW_LOADER=y
@@ -816,6 +822,7 @@
# CONFIG_SCSI_OSD_INITIATOR is not setCONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
+CONFIG_ATA_VERBOSE_ERROR=yCONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
# CONFIG_SATA_AHCI is not set
@@ -839,6 +846,7 @@
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
+# CONFIG_PATA_ATP867X is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
@@ -871,6 +879,7 @@
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_QDI is not set
# CONFIG_PATA_RADISYS is not set
+# CONFIG_PATA_RDC is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
@@ -973,6 +982,7 @@
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_KS8842 is not set
+# CONFIG_KS8851_MLL is not setCONFIG_VIA_RHINE=y
# CONFIG_VIA_RHINE_MMIO is not set
# CONFIG_SC92031 is not set
@@ -1005,12 +1015,7 @@
# CONFIG_JME is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
-
-#
-# Wireless LAN
-#
-# CONFIG_WLAN_PRE80211 is not set
-# CONFIG_WLAN_80211 is not set
+# CONFIG_WLAN is not set
#
# Enable WiMAX (Networking options) to see the WiMAX drivers @@ -1063,6 +1068,7 @@
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
+# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
@@ -1088,9 +1094,13 @@
# Input Device Drivers
#CONFIG_INPUT_KEYBOARD=y
+# CONFIG_KEYBOARD_ADP5588 is not setCONFIG_KEYBOARD_ATKBD=y
+# CONFIG_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
+# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_NEWTON is not set
+# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
@@ -1102,6 +1112,7 @@CONFIG_MOUSE_PS2_LIFEBOOK=yCONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
+# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
@@ -1125,6 +1136,7 @@CONFIG_INPUT_YEALINK=m
# CONFIG_INPUT_CM109 is not setCONFIG_INPUT_UINPUT=m
+# CONFIG_INPUT_WINBOND_CIR is not set
#
# Hardware I/O ports
@@ -1224,6 +1236,7 @@CONFIG_DEVPORT=yCONFIG_I2C=mCONFIG_I2C_BOARDINFO=y
+CONFIG_I2C_COMPAT=y
# CONFIG_I2C_CHARDEV is not setCONFIG_I2C_HELPER_AUTO=yCONFIG_I2C_ALGOBIT=m
@@ -1251,6 +1264,11 @@CONFIG_I2C_VIAPRO=m
#
+# ACPI drivers
+#
+# CONFIG_I2C_SCMI is not set
+
+#
# I2C system bus drivers (mostly embedded / system-on-chip) #
# CONFIG_I2C_OCORES is not set
@@ -1282,9 +1300,6 @@
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
-# CONFIG_SENSORS_PCF8574 is not set
-# CONFIG_PCF8575 is not set
-# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
@@ -1320,6 +1335,7 @@
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
+# CONFIG_SBC_FITPC2_WATCHDOG is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
@@ -1377,6 +1393,7 @@
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_WM8400 is not set
+# CONFIG_MFD_WM831X is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_AB3100_CORE is not set
@@ -1397,6 +1414,7 @@
# CONFIG_AGP_SWORKS is not setCONFIG_AGP_VIA=m
# CONFIG_AGP_EFFICEON is not set
+CONFIG_VGA_ARB=yCONFIG_DRM=m
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
@@ -1480,6 +1498,7 @@
# CONFIG_LOGO is not setCONFIG_SOUND=mCONFIG_SOUND_OSS_CORE=y +CONFIG_SOUND_OSS_CORE_PRECLAIM=yCONFIG_SND=mCONFIG_SND_TIMER=mCONFIG_SND_PCM=m
@@ -1501,6 +1520,7 @@
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not setCONFIG_SND_VMASTER=y
+CONFIG_SND_DMA_SGBUF=yCONFIG_SND_RAWMIDI_SEQ=m
# CONFIG_SND_OPL3_LIB_SEQ is not set
# CONFIG_SND_OPL4_LIB_SEQ is not set
@@ -1599,7 +1619,6 @@CONFIG_AC97_BUS=mCONFIG_HID_SUPPORT=yCONFIG_HID=m
-# CONFIG_HID_DEBUG is not setCONFIG_HIDRAW=y
#
@@ -1623,6 +1642,7 @@CONFIG_HID_EZKEY=mCONFIG_HID_KYE=mCONFIG_HID_GYRATION=m
+CONFIG_HID_TWINHAN=mCONFIG_HID_KENSINGTON=mCONFIG_HID_LOGITECH=m
# CONFIG_LOGITECH_FF is not set
@@ -1676,6 +1696,7 @@
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
+# CONFIG_USB_ISP1362_HCD is not setCONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
@@ -1855,8 +1876,8 @@
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
# CONFIG_EXT3_FS_XATTR is not setCONFIG_EXT4_FS=y
-# CONFIG_EXT4DEV_COMPAT is not set
# CONFIG_EXT4_FS_XATTR is not set
+# CONFIG_EXT4_DEBUG is not setCONFIG_JBD=yCONFIG_JBD2=y
# CONFIG_REISERFS_FS is not set
@@ -1865,6 +1886,7 @@
# CONFIG_XFS_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_BTRFS_FS is not set
+# CONFIG_NILFS2_FS is not setCONFIG_FILE_LOCKING=yCONFIG_FSNOTIFY=yCONFIG_DNOTIFY=y
@@ -2003,6 +2025,7 @@
# CONFIG_ENABLE_MUST_CHECK is not setCONFIG_FRAME_WARN=0CONFIG_MAGIC_SYSRQ=y
+CONFIG_STRIP_ASM_SYMS=yCONFIG_UNUSED_SYMBOLS=y
# CONFIG_DEBUG_FS is not set
# CONFIG_HEADERS_CHECK is not set
@@ -2039,6 +2062,7 @@
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
+# CONFIG_DEBUG_CREDENTIALS is not setCONFIG_ARCH_WANT_FRAME_POINTERS=y
# CONFIG_FRAME_POINTER is not set
# CONFIG_BOOT_PRINTK_DELAY is not set
@@ -2046,6 +2070,7 @@
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
+# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not setCONFIG_SYSCTL_SYSCALL_CHECK=y
@@ -2057,7 +2082,7 @@CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=yCONFIG_HAVE_DYNAMIC_FTRACE=yCONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
-CONFIG_HAVE_FTRACE_SYSCALLS=y
+CONFIG_HAVE_SYSCALL_TRACEPOINTS=yCONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
@@ -2118,7 +2143,7 @@CONFIG_CRYPTO_PCOMP=mCONFIG_CRYPTO_MANAGER=mCONFIG_CRYPTO_MANAGER2=m
-CONFIG_CRYPTO_GF128MUL=m
+CONFIG_CRYPTO_GF128MUL=yCONFIG_CRYPTO_NULL=mCONFIG_CRYPTO_WORKQUEUE=mCONFIG_CRYPTO_CRYPTD=m
@@ -2129,7 +2154,7 @@
# Authenticated Encryption with Associated Data
#CONFIG_CRYPTO_CCM=m
-CONFIG_CRYPTO_GCM=m
+# CONFIG_CRYPTO_GCM is not setCONFIG_CRYPTO_SEQIV=m
#
@@ -2148,12 +2173,14 @@
#CONFIG_CRYPTO_HMAC=mCONFIG_CRYPTO_XCBC=m
+CONFIG_CRYPTO_VMAC=m
#
# Digest
#CONFIG_CRYPTO_CRC32C=m
# CONFIG_CRYPTO_CRC32C_INTEL is not set
+# CONFIG_CRYPTO_GHASH is not setCONFIG_CRYPTO_MD4=mCONFIG_CRYPTO_MD5=mCONFIG_CRYPTO_MICHAEL_MIC=m
@@ -2208,7 +2235,6 @@
# CONFIG_CRYPTO_DEV_GEODE is not set
# CONFIG_CRYPTO_DEV_HIFN_795X is not setCONFIG_HAVE_KVM=y
-CONFIG_HAVE_KVM_IRQCHIP=y
# CONFIG_VIRTUALIZATION is not set
# CONFIG_BINARY_PRINTF is not set
Quoted 26 lines · Mark Lord
Quoted 25 lines · Mark Lord
Quoted 3 lines · Rafael J. Wysocki
Have you attached a (2.6.31.x) dmesg output from the failing box somewhere
or have I just missed it?..
The log from failing on 2.6.32.5 was linked to originally:
http://rtr.ca/2.6.32_page_alloc_fail_logs.txt
Here's the subsequent revert reboot back to 2.6.32.12:http://rtr.ca/2.6.31.12_ok_logs.txt
In the time available thus far, I haven't managed to get 2.6.32.5 to boot/run on anything else, using either a UP kernel, or "nosmp", along with "mem=512M".
With a bit of luck, I'll have another similar VIA C7 box delivered here by tonight,
and should be able to boot/run that off a cloned disk image from the original system.Then we can poke at things a bit more.
If the box works, it will be 100% at our disposal.
Quoted 9 lines
Quoted 4 lines
Note that it's far too slow for practical direct use of git-bisect, taking hours to rebuild its own kernels. It might be possible, with a fair bit of effort, to arrange a quad-core box to do the kernel compiles for it, though.
..
Rather than wasting time trying to bisect a full major kernel revision, I think instead I'll just focus on
mm/page_alloc.c.
Well, it might not eve be necessary. In the patch I sent you, it pointed the finger at commit 5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 being the problem in that case. I believe your problem is a variation of the slowdown-in-swapping problem except in your case it manifests as GFP_ATOMIC allocations failing.
If the fix does not help you, then I'll take a fresh look at the other commits with your particular problem in mind.
Quoted 423 lines
It appears the biggest changes were to
get_page_from_freelist(), so perhaps we'll just revert that one function and see how the clone system behaves.In the meanwhile, here's the diff -u of the two .config files:
linux/.config +2
-2@@ -1,7 +1,7 @@ # # Automatically generated make config: don't edit -# Linux kernel version: 2.6.31.12 -# Tue Jan 26 21:38:36 2010 +# Linux kernel version: 2.6.32.6 +# Tue Jan 26 20:57:44 2010 # # CONFIG_64BIT is not set
CONFIG_X86_32=ylinux/.config +0
-0@@ -16,7 +16,6 @@
CONFIG_LOCKDEP_SUPPORT=yCONFIG_STACKTRACE_SUPPORT=yCONFIG_HAVE_LATENCYTOP_SUPPORT=y
-CONFIG_FAST_CMPXCHG_LOCAL=yCONFIG_MMU=yCONFIG_ZONE_DMA=yCONFIG_GENERIC_ISA_DMA=ylinux/.config +0
-0@@ -33,7 +32,8 @@
CONFIG_ARCH_HAS_DEFAULT_IDLE=yCONFIG_ARCH_HAS_CACHE_LINE_SIZE=yCONFIG_HAVE_SETUP_PER_CPU_AREA=ylinux/.config +2
-1-CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y +CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y +CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y#
CONFIG_HAVE_CPUMASK_OF_CPU_MAPis not setCONFIG_ARCH_HIBERNATION_POSSIBLE=yCONFIG_ARCH_SUSPEND_POSSIBLE=ylinux/.config +5
-4@@ -76,11 +76,12 @@ # # RCU Subsystem # -CONFIG_CLASSIC_RCU=y -# CONFIG_TREE_RCU is not set -# CONFIG_PREEMPT_RCU is not set +CONFIG_TREE_RCU=y +# CONFIG_TREE_PREEMPT_RCU is not set +# CONFIG_RCU_TRACE is not set +CONFIG_RCU_FANOUT=8 +# CONFIG_RCU_FANOUT_EXACT is not set # CONFIG_TREE_RCU_TRACE is not set -# CONFIG_PREEMPT_RCU_TRACE is not set
CONFIG_IKCONFIG=yCONFIG_IKCONFIG_PROC=yCONFIG_LOG_BUF_SHIFT=15linux/.config +0
-0@@ -123,21 +124,20 @@
CONFIG_EVENTFD=yCONFIG_SHMEM=yCONFIG_AIO=ylinux/.config +1
-1-CONFIG_HAVE_PERF_COUNTERS=y +CONFIG_HAVE_PERF_EVENTS=y#
linux/.config +1
-1-# Performance Counters +# Kernel Performance Events And Counters#
+#CONFIG_PERF_EVENTSis not set
#CONFIG_PERF_COUNTERSis not setCONFIG_VM_EVENT_COUNTERS=yCONFIG_PCI_QUIRKS=y
-CONFIG_STRIP_ASM_SYMS=yCONFIG_COMPAT_BRK=yCONFIG_SLAB=y
#CONFIG_SLUBis not set
#CONFIG_SLOBis not set
#CONFIG_PROFILINGis not set
-#CONFIG_MARKERSis not setCONFIG_HAVE_OPROFILE=y
#CONFIG_KPROBESis not setCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=ylinux/.config +1
-1@@ -151,7 +151,7 @@ # # GCOV-based kernel profiling # -# CONFIG_SLOW_WORK is not set +CONFIG_SLOW_WORK=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=yCONFIG_SLABINFO=yCONFIG_RT_MUTEXES=ylinux/.config +1
-0@@ -217,6 +217,7 @@ # CONFIG_MVIAC7 is not set # CONFIG_MPSC is not set # CONFIG_MCORE2 is not set +# CONFIG_MATOM is not set # CONFIG_GENERIC_CPU is not set # CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=ylinux/.config +0
-1@@ -250,7 +251,6 @@ # CONFIG_PREEMPT is not set # CONFIG_X86_UP_APIC is not set # CONFIG_X86_MCE is not set -# CONFIG_X86_ANCIENT_MCE is not set
CONFIG_VM86=y
#CONFIG_TOSHIBAis not set
#CONFIG_I8Kis not setlinux/.config +0
-0@@ -283,6 +283,7 @@
CONFIG_VIRT_TO_BUS=yCONFIG_HAVE_MLOCK=yCONFIG_HAVE_MLOCKED_PAGE_BIT=y
+#CONFIG_KSMis not setCONFIG_DEFAULT_MMAP_MIN_ADDR=4096
#CONFIG_X86_CHECK_BIOS_CORRUPTIONis not set
#CONFIG_X86_RESERVE_LOW_64Kis not setlinux/.config +1
-0@@ -313,6 +314,7 @@ # CONFIG_PM_DEBUG is not set # CONFIG_SUSPEND is not set # CONFIG_HIBERNATION is not set +# CONFIG_PM_RUNTIME is not set
CONFIG_ACPI=yCONFIG_ACPI_PROCFS=yCONFIG_ACPI_PROCFS_POWER=ylinux/.config +0
-0@@ -324,6 +326,7 @@
CONFIG_ACPI_FAN=mCONFIG_ACPI_DOCK=yCONFIG_ACPI_PROCESSOR=m
+#CONFIG_ACPI_PROCESSOR_AGGREGATORis not setCONFIG_ACPI_THERMAL=m
#CONFIG_ACPI_CUSTOM_DSDTis not setCONFIG_ACPI_BLACKLIST_YEAR=0linux/.config +0
-0@@ -332,6 +335,7 @@
CONFIG_X86_PM_TIMER=yCONFIG_ACPI_CONTAINER=m
#CONFIG_ACPI_SBSis not set
+#CONFIG_SFIis not set#
# CPU Frequency scalinglinux/.config +1
-0@@ -626,6 +630,7 @@ # # CONFIG_IP_DCCP_DEBUG is not set # CONFIG_IP_SCTP is not set +# CONFIG_RDS is not set # CONFIG_TIPC is not set # CONFIG_ATM is not set
CONFIG_STP=mlinux/.config +0
-0@@ -723,6 +728,7 @@ # Generic Driver Options #
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
+#CONFIG_DEVTMPFSis not setCONFIG_STANDALONE=yCONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_FW_LOADER=ylinux/.config +0
-0@@ -816,6 +822,7 @@ # CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
#CONFIG_ATA_NONSTANDARDis not set
+CONFIG_ATA_VERBOSE_ERROR=yCONFIG_ATA_ACPI=y
#CONFIG_SATA_PMPis not set
#CONFIG_SATA_AHCIis not setlinux/.config +3
-0@@ -839,6 +846,7 @@ # CONFIG_PATA_ALI is not set # CONFIG_PATA_AMD is not set # CONFIG_PATA_ARTOP is not set +# CONFIG_PATA_ATP867X is not set # CONFIG_PATA_ATIIXP is not set # CONFIG_PATA_CMD640_PCI is not set # CONFIG_PATA_CMD64X is not set@@ -871,6 +879,7 @@ # CONFIG_PATA_PDC_OLD is not set # CONFIG_PATA_QDI is not set # CONFIG_PATA_RADISYS is not set +# CONFIG_PATA_RDC is not set # CONFIG_PATA_RZ1000 is not set # CONFIG_PATA_SC1200 is not set # CONFIG_PATA_SERVERWORKS is not set@@ -973,6 +982,7 @@ # CONFIG_SUNDANCE is not set # CONFIG_TLAN is not set # CONFIG_KS8842 is not set +# CONFIG_KS8851_MLL is not set
CONFIG_VIA_RHINE=y
#CONFIG_VIA_RHINE_MMIOis not set
#CONFIG_SC92031is not setlinux/.config +2
-6@@ -1005,12 +1015,7 @@ # CONFIG_JME is not set # CONFIG_NETDEV_10000 is not set # CONFIG_TR is not set - -# -# Wireless LAN -# -# CONFIG_WLAN_PRE80211 is not set -# CONFIG_WLAN_80211 is not set +# CONFIG_WLAN is not set # # Enable WiMAX (Networking options) to see the WiMAX drivers@@ -1063,6 +1068,7 @@ # CONFIG_NETCONSOLE is not set # CONFIG_NETPOLL is not set # CONFIG_NET_POLL_CONTROLLER is not set +# CONFIG_VMXNET3 is not set # CONFIG_ISDN is not set # CONFIG_PHONE is not set@@ -1088,9 +1094,13 @@ # Input Device Drivers #
CONFIG_INPUT_KEYBOARD=y
+#CONFIG_KEYBOARD_ADP5588is not setCONFIG_KEYBOARD_ATKBD=y
+#CONFIG_QT2160is not set
#CONFIG_KEYBOARD_LKKBDis not set
+#CONFIG_KEYBOARD_MAX7359is not set
#CONFIG_KEYBOARD_NEWTONis not set
+#CONFIG_KEYBOARD_OPENCORESis not set
#CONFIG_KEYBOARD_STOWAWAYis not set
#CONFIG_KEYBOARD_SUNKBDis not set
#CONFIG_KEYBOARD_XTKBDis not setlinux/.config +0
-0@@ -1102,6 +1112,7 @@
CONFIG_MOUSE_PS2_LIFEBOOK=yCONFIG_MOUSE_PS2_TRACKPOINT=y
#CONFIG_MOUSE_PS2_ELANTECHis not set
+#CONFIG_MOUSE_PS2_SENTELICis not set
#CONFIG_MOUSE_PS2_TOUCHKITis not set
#CONFIG_MOUSE_SERIALis not set
#CONFIG_MOUSE_APPLETOUCHis not setlinux/.config +0
-0@@ -1125,6 +1136,7 @@
CONFIG_INPUT_YEALINK=m
#CONFIG_INPUT_CM109is not setCONFIG_INPUT_UINPUT=m
+#CONFIG_INPUT_WINBOND_CIRis not set#
# Hardware I/O portslinux/.config +0
-0@@ -1224,6 +1236,7 @@
CONFIG_DEVPORT=yCONFIG_I2C=mCONFIG_I2C_BOARDINFO=y
+CONFIG_I2C_COMPAT=y
#CONFIG_I2C_CHARDEVis not setCONFIG_I2C_HELPER_AUTO=yCONFIG_I2C_ALGOBIT=mlinux/.config +0
-0@@ -1251,6 +1264,11 @@
CONFIG_I2C_VIAPRO=m#
linux/.config +5
-0+# ACPI drivers +# +# CONFIG_I2C_SCMI is not set + +## I2C system bus drivers (mostly embedded / system-on-chip) #
#CONFIG_I2C_OCORESis not setlinux/.config +2
-3@@ -1282,9 +1300,6 @@ # Miscellaneous I2C Chip support # # CONFIG_DS1682 is not set -# CONFIG_SENSORS_PCF8574 is not set -# CONFIG_PCF8575 is not set -# CONFIG_SENSORS_PCA9539 is not set # CONFIG_SENSORS_TSL2550 is not set # CONFIG_I2C_DEBUG_CORE is not set # CONFIG_I2C_DEBUG_ALGO is not set@@ -1320,6 +1335,7 @@ # CONFIG_ALIM1535_WDT is not set # CONFIG_ALIM7101_WDT is not set # CONFIG_SC520_WDT is not set +# CONFIG_SBC_FITPC2_WATCHDOG is not set # CONFIG_EUROTECH_WDT is not set # CONFIG_IB700_WDT is not set # CONFIG_IBMASR is not set@@ -1377,6 +1393,7 @@ # CONFIG_HTC_PASIC3 is not set # CONFIG_MFD_TMIO is not set # CONFIG_MFD_WM8400 is not set +# CONFIG_MFD_WM831X is not set # CONFIG_MFD_WM8350_I2C is not set # CONFIG_MFD_PCF50633 is not set # CONFIG_AB3100_CORE is not set@@ -1397,6 +1414,7 @@ # CONFIG_AGP_SWORKS is not set
CONFIG_AGP_VIA=m
#CONFIG_AGP_EFFICEONis not set
+CONFIG_VGA_ARB=yCONFIG_DRM=m
#CONFIG_DRM_TDFXis not set
#CONFIG_DRM_R128is not setlinux/.config +0
-0@@ -1480,6 +1498,7 @@ # CONFIG_LOGO is not set
CONFIG_SOUND=mCONFIG_SOUND_OSS_CORE=y
+CONFIG_SOUND_OSS_CORE_PRECLAIM=yCONFIG_SND=mCONFIG_SND_TIMER=mCONFIG_SND_PCM=mlinux/.config +0
-0@@ -1501,6 +1520,7 @@ # CONFIG_SND_VERBOSE_PRINTK is not set # CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
+CONFIG_SND_DMA_SGBUF=yCONFIG_SND_RAWMIDI_SEQ=m
#CONFIG_SND_OPL3_LIB_SEQis not set
#CONFIG_SND_OPL4_LIB_SEQis not setlinux/.config +0
-0@@ -1599,7 +1619,6 @@
CONFIG_AC97_BUS=mCONFIG_HID_SUPPORT=yCONFIG_HID=m
-#CONFIG_HID_DEBUGis not setCONFIG_HIDRAW=y#
linux/.config +0
-0@@ -1623,6 +1642,7 @@
CONFIG_HID_EZKEY=mCONFIG_HID_KYE=mCONFIG_HID_GYRATION=m
+CONFIG_HID_TWINHAN=mCONFIG_HID_KENSINGTON=mCONFIG_HID_LOGITECH=m
#CONFIG_LOGITECH_FFis not setlinux/.config +1
-0@@ -1676,6 +1696,7 @@ # CONFIG_USB_OXU210HP_HCD is not set # CONFIG_USB_ISP116X_HCD is not set # CONFIG_USB_ISP1760_HCD is not set +# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=m
#CONFIG_USB_OHCI_BIG_ENDIAN_DESCis not set
#CONFIG_USB_OHCI_BIG_ENDIAN_MMIOis not setlinux/.config +0
-0@@ -1855,8 +1876,8 @@ # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set # CONFIG_EXT3_FS_XATTR is not set
CONFIG_EXT4_FS=y
-#CONFIG_EXT4DEV_COMPATis not set
#CONFIG_EXT4_FS_XATTRis not set
+#CONFIG_EXT4_DEBUGis not setCONFIG_JBD=yCONFIG_JBD2=y
#CONFIG_REISERFS_FSis not setlinux/.config +1
-0@@ -1865,6 +1886,7 @@ # CONFIG_XFS_FS is not set # CONFIG_OCFS2_FS is not set # CONFIG_BTRFS_FS is not set +# CONFIG_NILFS2_FS is not set
CONFIG_FILE_LOCKING=yCONFIG_FSNOTIFY=yCONFIG_DNOTIFY=ylinux/.config +0
-0@@ -2003,6 +2025,7 @@ # CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=0CONFIG_MAGIC_SYSRQ=y
+CONFIG_STRIP_ASM_SYMS=yCONFIG_UNUSED_SYMBOLS=y
#CONFIG_DEBUG_FSis not set
#CONFIG_HEADERS_CHECKis not setlinux/.config +1
-0@@ -2039,6 +2062,7 @@ # CONFIG_DEBUG_LIST is not set # CONFIG_DEBUG_SG is not set # CONFIG_DEBUG_NOTIFIERS is not set +# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
#CONFIG_FRAME_POINTERis not set
#CONFIG_BOOT_PRINTK_DELAYis not setlinux/.config +1
-0@@ -2046,6 +2070,7 @@ # CONFIG_RCU_CPU_STALL_DETECTOR is not set # CONFIG_BACKTRACE_SELF_TEST is not set # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set +# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set # CONFIG_FAULT_INJECTION is not set # CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=ylinux/.config +0
-0@@ -2057,7 +2082,7 @@
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=yCONFIG_HAVE_DYNAMIC_FTRACE=yCONFIG_HAVE_FTRACE_MCOUNT_RECORD=ylinux/.config +1
-1-CONFIG_HAVE_FTRACE_SYSCALLS=y +CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_TRACING_SUPPORT=y
#CONFIG_FTRACEis not set
#CONFIG_PROVIDE_OHCI1394_DMA_INITis not setlinux/.config +0
-0@@ -2118,7 +2143,7 @@
CONFIG_CRYPTO_PCOMP=mCONFIG_CRYPTO_MANAGER=mCONFIG_CRYPTO_MANAGER2=mlinux/.config +1
-1-CONFIG_CRYPTO_GF128MUL=m +CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=mCONFIG_CRYPTO_WORKQUEUE=mCONFIG_CRYPTO_CRYPTD=mlinux/.config +0
-0@@ -2129,7 +2154,7 @@ # Authenticated Encryption with Associated Data #
CONFIG_CRYPTO_CCM=mlinux/.config +1
-1-CONFIG_CRYPTO_GCM=m +# CONFIG_CRYPTO_GCM is not set
CONFIG_CRYPTO_SEQIV=m#
linux/.config +0
-0@@ -2148,12 +2173,14 @@ #
CONFIG_CRYPTO_HMAC=mCONFIG_CRYPTO_XCBC=m
+CONFIG_CRYPTO_VMAC=m#
# Digest
#CONFIG_CRYPTO_CRC32C=m
#CONFIG_CRYPTO_CRC32C_INTELis not set
+#CONFIG_CRYPTO_GHASHis not setCONFIG_CRYPTO_MD4=mCONFIG_CRYPTO_MD5=mCONFIG_CRYPTO_MICHAEL_MIC=mlinux/.config +0
-0@@ -2208,7 +2235,6 @@ # CONFIG_CRYPTO_DEV_GEODE is not set # CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
-CONFIG_HAVE_KVM_IRQCHIP=y
#CONFIG_VIRTUALIZATIONis not set
#CONFIG_BINARY_PRINTFis not set
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 1 line · Mel Gorman
On Thu, Jan 28, 2010 at 09:17:17AM -0500, Mark Lord wrote:
..
Quoted 2 lines
Quoted 2 lines
Rather than wasting time trying to bisect a full major kernel revision, I think instead I'll just focus on
mm/page_alloc.c.
..
Quoted 8 lines
Well, it might not eve be necessary. In the patch I sent you, it pointed the finger at commit
5f8dcc21211a3d4e3a7a5ca366b469fb88117f61being the problem in that case. I believe your problem is a variation of the slowdown-in-swapping problem except in your case it manifests asGFP_ATOMICallocations failing.If the fix does not help you, then I'll take a fresh look at the other commits with your particular problem in mind.
..
Last night, I installed 2.6.32.7, plus the patch you sent. So far, no allocation faults.
I'll leave it running for another day or so, and then perhaps revert the one patch to see which of the two things (new kernel, or patch) is responsible for the difference.
The changelog for 2.6.32.7 included something to fix default rsize/wsize values on NFS. Dunno if this might have had an effect or not, but when it was failing.. NFS (order 1) was the most frequent case.
-ml
Quoted 12 lines · Mark Lord
Quoted 1 line · Mel Gorman
On Thu, Jan 28, 2010 at 09:17:17AM -0500, Mark Lord wrote:
..
Quoted 2 lines
Quoted 2 lines
Rather than wasting time trying to bisect a full major kernel revision, I think instead I'll just focus on
mm/page_alloc.c...
Quoted 6 lines
Well, it might not eve be necessary. In the patch I sent you, it pointed the finger at commit
5f8dcc21211a3d4e3a7a5ca366b469fb88117f61being the problem in that case. I believe your problem is a variation of the slowdown-in-swapping problem except in your case it manifests asGFP_ATOMICallocations failing.
Quoted 7 lines
Quoted 2 lines
If the fix does not help you, then I'll take a fresh look at the other commits with your particular problem in mind.
..
Last night, I installed 2.6.32.7, plus the patch you sent. So far, no allocation faults.
Nice one.
Quoted 4 lines
I'll leave it running for another day or so, and then perhaps revert the one patch to see which of the two things (new kernel, or patch) is responsible for the difference.
Thanks, I'd appreciate it. While I'm reasonably confident the problem is with MIGRATE_RESERVE not being free as intended and that the patch fixes it, I'd like more proof.
Quoted 4 lines
The changelog for 2.6.32.7 included something to fix default rsize/wsize values on NFS. Dunno if this might have had an effect or not, but when it was failing.. NFS (order 1) was the most frequent case.
I consider it unlikely that a change in NFS is responsible. The network MTU would remain the same and it's a bigger factor in the size of allocations made. I think it's just a co-incidence that NFS reported the most failures simply because it was responsible for the bulk of network traffic.
Hugh, can I get a signed-off-by on that patch please? I can improve the changelog if you like and send it to Andrew for merging if you like.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 10 lines · Mel Gorman
Quoted 4 lines · Mark Lord
I'll leave it running for another day or so, and then perhaps revert the one patch to see which of the two things (new kernel, or patch) is responsible for the difference.
Thanks, I'd appreciate it. While I'm reasonably confident the problem is with
MIGRATE_RESERVEnot being free as intended and that the patch fixes it, I'd like more proof.
You're more confident about that than I am! It will be very satisfying if my patch turns out to make the difference, but still surprising to me.
Thank you for taking the time on this, Mark: I too would appreciate it if you could later determine whether it's new kernel or patch solving it.
Quoted 3 lines
Hugh, can I get a signed-off-by on that patch please? I can improve the changelog if you like and send it to Andrew for merging if you like.
I was adjusting the changelog and about to send direct to Linus Cc stable in a few minutes, since I'm guessing there might be a 33-rc6 today, which would be a pity to miss.
Whatever my reluctance to assume it's the fix to Mark's problem (which I'm not mentioning in the changelog), we are both sure it's a valid bugfix.
Hugh
Quoted 8 lines · Hugh Dickins
Quoted 7 lines · Mel Gorman
Quoted 4 lines · Mark Lord
I'll leave it running for another day or so, and then perhaps revert the one patch to see which of the two things (new kernel, or patch) is responsible for the difference.
Quoted 7 lines
Quoted 3 lines
Thanks, I'd appreciate it. While I'm reasonably confident the problem is with
MIGRATE_RESERVEnot being free as intended and that the patch fixes it, I'd like more proof.You're more confident about that than I am! It will be very satisfying if my patch turns out to make the difference, but still surprising to me.
My confidence/delusion is in part due to the reasons MIGRATE_RESERVE exists in the first place. Specifically, certain wireless network drivers were doing GFP_ATOMIC order-2 allocation a lot and failing miserably when anti-fragmentation was first introduced. The problem came down to a property of the buddy allocator implementation that kept min_free_kbytes worth of pages free at the lower addresses of the zone and this is where order-2 allocations were being made from and quickly freed meaning the area was generally available for a lot of time. MIGRATE_RESERVE was introduced to preserve that property of the buddy allocator and the allocation failure problems went away.
Mark's problem feels very similar to the wireless network drivers problem.
Maybe I'm deluding myself.
Quoted 11 lines
Thank you for taking the time on this, Mark: I too would appreciate it if you could later determine whether it's new kernel or patch solving it.
Quoted 3 lines
Hugh, can I get a signed-off-by on that patch please? I can improve the changelog if you like and send it to Andrew for merging if you like.
I was adjusting the changelog and about to send direct to Linus Cc stable in a few minutes, since I'm guessing there might be a 33-rc6 today, which would be a pity to miss.
Great.
Quoted 3 lines
Whatever my reluctance to assume it's the fix to Mark's problem (which I'm not mentioning in the changelog), we are both sure it's a valid bugfix.
Indeed. Thanks.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
PATCH 1/1 mm: fix migratetype bug which slowed swapping
After memory pressure has forced it to dip into the reserves, 2.6.32's 5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 "page-allocator: split per-cpu list into one-list-per-migrate-type" has been returning MIGRATE_RESERVE pages to the MIGRATE_MOVABLE free_list: in some sense depleting reserves.
Fix that in the most straightforward way (which, considering the overheads of alternative approaches, is Mel's preference): the right migratetype is already in page_private(page), but free_pcppages_bulk() wasn't using it.
How did this bug show up? As a 20% slowdown in my tmpfs loop kbuild swapping tests, on PowerMac G5 with SLUB allocator. Bisecting to that commit was easy, but explaining the magnitude of the slowdown not easy.
The same effect appears, but much less markedly, with SLAB, and even less markedly on other machines (the PowerMac divides into fewer zones than x86, I think that may be a factor). We guess that lumpy reclaim of short-lived high-order pages is implicated in some way, and probably this bug has been tickling a poor decision somewhere in page reclaim.
But instrumentation hasn't told me much, I've run out of time and imagination to determine exactly what's going on, and shouldn't hold up the fix any longer: it's valid, and might even fix other misbehaviours.
---
mm/page_alloc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/page_alloc.c +3 -2
@@ -556,8 +556,9 @@ static void free_pcppages_bulk(struct zo page = list_entry(list->prev, struct page, lru);/* must delete as __free_one_page list manipulates */ list_del (&page->lru); -__free_one_page (page, zone,0 , migratetype); -trace_mm_page_pcpu_drain (page,0 , migratetype); +/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */ +__free_one_page (page, zone,0 ,page_private (page)); +trace_mm_page_pcpu_drain (page,0 ,page_private (page)); }while (--count && --batch_free && !list_empty (list)); }spin_unlock (&zone->lock);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>