Skip to contentSkip to conversation

HIGHMEM64G Kernel (2.6.23.1) makes system crawl

PATCH Rajkumar S 4 participants 6 messages Oct 24, 2007

lore
Share & tools
lore

Rajkumar S

author
#0 lore raw

Hello,

I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with 4GB Ram, running Debian Lenny.

Since the box has 4 GB ram I compiled a big mem kernel, but the machine is very slow while running big mem kernel. It takes about 37 minutes to compile the intel e1000 driver (e1000-7.6.5.tar.gz) from intel site. But it's performing normally when using a non big mem kernel. The diff of the .config between working and non working is as follows.

config-bigmem 2007-10-24 12:48:56.000000000 +0530 renamed from config-nobigmem 2007-10-24 12:51:42.000000000 +0530 +9 -3
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.23.1
-# Wed Oct 24 12:50:21 2007
+# Tue Oct 23 18:08:30 2007
 #
 CONFIG_X86_32=y
 CONFIG_GENERIC_TIME=y
@@ -189,10 +189,11 @@
 CONFIG_DELL_RBU=m
 CONFIG_DCDBAS=m
 CONFIG_DMIID=y
-CONFIG_NOHIGHMEM=y
+# CONFIG_NOHIGHMEM is not set
 # CONFIG_HIGHMEM4G is not set
-# CONFIG_HIGHMEM64G is not set
+CONFIG_HIGHMEM64G=y
 CONFIG_PAGE_OFFSET=0xC0000000
+CONFIG_HIGHMEM=y
 CONFIG_X86_PAE=y
 CONFIG_ARCH_FLATMEM_ENABLE=y
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
@@ -211,6 +212,7 @@
 CONFIG_BOUNCE=y
 CONFIG_NR_QUICK=1
 CONFIG_VIRT_TO_BUS=y
+# CONFIG_HIGHPTE is not set
 # CONFIG_MATH_EMULATION is not set
 CONFIG_MTRR=y
 CONFIG_EFI=y
@@ -223,11 +225,13 @@
 # CONFIG_HZ_1000 is not set
 CONFIG_HZ=250
 CONFIG_KEXEC=y
+# CONFIG_CRASH_DUMP is not set
 CONFIG_PHYSICAL_START=0x100000
 # CONFIG_RELOCATABLE is not set
 CONFIG_PHYSICAL_ALIGN=0x100000
 CONFIG_HOTPLUG_CPU=y
 CONFIG_COMPAT_VDSO=y
+CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
 
 #
 # Power management options (ACPI, APM)
@@ -1283,6 +1287,7 @@
 CONFIG_I2O=m
 CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
 CONFIG_I2O_EXT_ADAPTEC=y
+CONFIG_I2O_EXT_ADAPTEC_DMA64=y
 CONFIG_I2O_CONFIG=m
 CONFIG_I2O_CONFIG_OLD_IOCTL=y
 CONFIG_I2O_BUS=m
@@ -3323,6 +3328,7 @@
 # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
 # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
 # CONFIG_DEBUG_KOBJECT is not set
+# CONFIG_DEBUG_HIGHMEM is not set
 CONFIG_DEBUG_BUGVERBOSE=y
 # CONFIG_DEBUG_INFO is not set
 # CONFIG_DEBUG_VM is not set

I was having this same issue with debian kernel also (2.6.22-2-686), ie stock kernel will work, but big mem is slow. Which is why I tried latest kernel.

I am posting the full .config at http://pastebin.ca/747758 dmesg while booting big mem kernel is at http://pastebin.ca/747766

raj

#1 lore raw
Quoted 76 lines · Rajkumar S

Hello,

I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with 4GB Ram, running Debian Lenny.

Since the box has 4 GB ram I compiled a big mem kernel, but the machine is very slow while running big mem kernel. It takes about 37 minutes to compile the intel e1000 driver (e1000-7.6.5.tar.gz) from intel site. But it's performing normally when using a non big mem kernel. The diff of the .config between working and non working is as follows.

config-bigmem 2007-10-24 12:48:56.000000000 +0530 renamed from config-nobigmem 2007-10-24 12:51:42.000000000 +0530 +9 -3
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.23.1
-# Wed Oct 24 12:50:21 2007
+# Tue Oct 23 18:08:30 2007
 #
 CONFIG_X86_32=y
 CONFIG_GENERIC_TIME=y
@@ -189,10 +189,11 @@
 CONFIG_DELL_RBU=m
 CONFIG_DCDBAS=m
 CONFIG_DMIID=y
-CONFIG_NOHIGHMEM=y
+# CONFIG_NOHIGHMEM is not set
 # CONFIG_HIGHMEM4G is not set
-# CONFIG_HIGHMEM64G is not set
+CONFIG_HIGHMEM64G=y
 CONFIG_PAGE_OFFSET=0xC0000000
+CONFIG_HIGHMEM=y
 CONFIG_X86_PAE=y
 CONFIG_ARCH_FLATMEM_ENABLE=y
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
@@ -211,6 +212,7 @@
 CONFIG_BOUNCE=y
 CONFIG_NR_QUICK=1
 CONFIG_VIRT_TO_BUS=y
+# CONFIG_HIGHPTE is not set
 # CONFIG_MATH_EMULATION is not set
 CONFIG_MTRR=y
 CONFIG_EFI=y
@@ -223,11 +225,13 @@
 # CONFIG_HZ_1000 is not set
 CONFIG_HZ=250
 CONFIG_KEXEC=y
+# CONFIG_CRASH_DUMP is not set
 CONFIG_PHYSICAL_START=0x100000
 # CONFIG_RELOCATABLE is not set
 CONFIG_PHYSICAL_ALIGN=0x100000
 CONFIG_HOTPLUG_CPU=y
 CONFIG_COMPAT_VDSO=y
+CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
 
 #
 # Power management options (ACPI, APM)
@@ -1283,6 +1287,7 @@
 CONFIG_I2O=m
 CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
 CONFIG_I2O_EXT_ADAPTEC=y
+CONFIG_I2O_EXT_ADAPTEC_DMA64=y
 CONFIG_I2O_CONFIG=m
 CONFIG_I2O_CONFIG_OLD_IOCTL=y
 CONFIG_I2O_BUS=m
@@ -3323,6 +3328,7 @@
 # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
 # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
 # CONFIG_DEBUG_KOBJECT is not set
+# CONFIG_DEBUG_HIGHMEM is not set
 CONFIG_DEBUG_BUGVERBOSE=y
 # CONFIG_DEBUG_INFO is not set
 # CONFIG_DEBUG_VM is not set
Quoted 8 lines

I was having this same issue with debian kernel also (2.6.22-2-686), ie stock kernel will work, but big mem is slow. Which is why I tried latest kernel.

I am posting the full .config at http://pastebin.ca/747758 dmesg while booting big mem kernel is at http://pastebin.ca/747766

raj

I thought that for 4GB of memory the CONFIG_HIGHMEM4G=y CONFIG_HIGHMEM64G is not set would be enough.

Quoted 1 line

From looking in source code the CONFIG_HIGHMEM64G is broken in many respects,

in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even run on None 64-bit machines? Not many!

You have a magnificent 64-bit Machine, why?

Boaz

#2 lore raw
Quoted 8 lines · Boaz Harrosh

I thought that for 4GB of memory the CONFIG_HIGHMEM4G=y CONFIG_HIGHMEM64G is not set would be enough.

From looking in source code the CONFIG_HIGHMEM64G is broken in many respects, in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even run on None 64-bit machines? Not many!

You have a magnificent 64-bit Machine, why?

CONFIG_HIGHMEM4G means 4GB address space. Since PCI and BIOS use some of the first 4GB address space, a machine with 4GB of actual RAM must map some of that RAM above the 4GB address space, so you access it all you need to enable PAE (address space extensions intel invented to give 36bit addressing on 32bit machines), which lets the OS map memory above 4GB (up to 64GB), to let applications use it. Applications are still limited to a 32bit address space. 64bit machines of course just use all memory directly without any silly extensions.

The common intel BIOS bug relating to configuring the MTRR and such for caching of memory makes the top portion of memory uncached and hence very slow, which means any code placed in the top part of memory gets slow. The kernel likes to place itself at the top of memory which means the kernel gets slow and hence the whole system gets slow. Holefully intel will learn to make proper BIOSes real soon so that people with intel boards can stop having such slow machines when they have lots of ram.

--
Len Sorensen

#3 lore raw
Quoted 28 lines · On Wed, Oct 24 2007 at 18:35 +0200, [email protected] (Lennart…
Quoted 8 lines · Boaz Harrosh

I thought that for 4GB of memory the CONFIG_HIGHMEM4G=y CONFIG_HIGHMEM64G is not set would be enough.

From looking in source code the CONFIG_HIGHMEM64G is broken in many respects, in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even run on None 64-bit machines? Not many!

You have a magnificent 64-bit Machine, why?

CONFIG_HIGHMEM4G means 4GB address space. Since PCI and BIOS use some of the first 4GB address space, a machine with 4GB of actual RAM must map some of that RAM above the 4GB address space, so you access it all you need to enable PAE (address space extensions intel invented to give 36bit addressing on 32bit machines), which lets the OS map memory above 4GB (up to 64GB), to let applications use it. Applications are still limited to a 32bit address space. 64bit machines of course just use all memory directly without any silly extensions.

The common intel BIOS bug relating to configuring the MTRR and such for caching of memory makes the top portion of memory uncached and hence very slow, which means any code placed in the top part of memory gets slow. The kernel likes to place itself at the top of memory which means the kernel gets slow and hence the whole system gets slow. Holefully intel will learn to make proper BIOSes real soon so that people with intel boards can stop having such slow machines when they have lots of ram.

Quoted 2 lines

--
Len Sorensen

So one thing I don't understand is the difference between CONFIG_HIGHMEM4G and regular 32-bit. I always thought that 32 bit means 4GB address space and that the HIGHMEM4G is using your above trick but with 32-bit dma_addr_t since there is only actual 4GB of real memory and the rest is mapping tricks. But I guess I was wrong.

The other point I was making is why use 36 bit trickery when you have a machine that is capable of the full 64-bit, and runs much faster at that? Just a waste of a good machine, wont you say? Perhaps we learn to use x86_64 ARCHs before Intel fixes their BIOS, and they where right all along?

Boaz

#4 lore raw
Quoted 5 lines · Boaz Harrosh

So one thing I don't understand is the difference between CONFIG_HIGHMEM4G and regular 32-bit. I always thought that 32 bit means 4GB address space and that the HIGHMEM4G is using your above trick but with 32-bit dma_addr_t since there is only actual 4GB of real memory and the rest is mapping tricks. But I guess I was wrong.

For efficiency reasons the kernel only supports about 900MB of ram on i386, unless you enable the support for a full 4GB address space which is what CONFIG_HIGHMEM4G does. What exactly changes in the memory management and mapping I am not sure of, but it made some difference. It only gives 4GB of address space and does not enable PAE, which means if the system has more ram than it can map in the first 4GB of address space, then the remaining ram is simply invisible to the system. So you might have 4GB ram, and only see 3GB or 2.5GB of it with a 4GB kernel. If you want the kernel to have access to the rest of the ram then you need a kernel capable of addressing physical ram at addresses higher than 4GB, which means either PAE (CONFIG_HIGHMEM64G) or a 64bit kernel. I believe the comments for the CONFIG_HIGHMEM* options was recently updated to me correct rather than the rather misleading messages they have had for years. Unfortunately it would appear those nice updates have not gone tp Linus' tree so we are still stuck with the old awful descriptions. I wonder what happened to them.

Quoted 4 lines

The other point I was making is why use 36 bit trickery when you have a machine that is capable of the full 64-bit, and runs much faster at that? Just a waste of a good machine, wont you say? Perhaps we learn to use x86_64 ARCHs before Intel fixes their BIOS, and they where right all along?

Because the 36bit trickery works with an i386 system and kernel. Not everyone wants to run 64bit and deal with the (few) programs that don't yet work on 64bit systems.

Personally I run a 64bit kernel with a 32bit user space on any 64bit capable machines I deal with. That seems to work fine, and it lets me play with 64bit programs in a 64bit chroot when I want to.

--
Len Sorensen

#5 lore raw
In reply to Boaz Harrosh
Quoted 5 lines · Boaz Harrosh

From looking in source code the CONFIG_HIGHMEM64G is broken in many respects, in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even run on None 64-bit machines? Not many!

Total and utter bullsh*t. It runs on virtually all modern x86's since the P6/K7 (certain Pentium M chips being an exception).

Also, it is required for NX protection and for several of the paravirt environments to work.

-hpa