Skip to contentSkip to conversation

1-2-3 GB

PATCH In review [email protected] 12 participants 26 messages Jan 12, 2002 – Jan 19, 2002

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 1-2-3 GB

Patch to have 1-3 GB of virtual memory and not show up as highmem:

Tested on uniprocessor Athlon with 1024 MB RAM and 1027 MB swap. Caused no LTP (ltp-20020108) regressions.
Did a test like http://marc.theaimsgroup.com/?l=linux-kernel&m=101064072924424&w=2
This time the test completed in 51 minutes (11% faster) and I had setiathome running the whole time and listened to 12 mp3s sampled at 128k.

dmesg|grep Mem
Memory: 1029848k/1048512k available (1054k kernel code, 18276k reserved, 260k data, 240k init, 0k highmem)

egrep '^CONFIG_HIGH|GB' /usr/src/linux/.config
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
# CONFIG_1GB is not set
CONFIG_2GB=y
# CONFIG_3GB is not set
# CONFIG_05GB is not set

uname -a
Linux rushmore 2.4.18pre2aa2-2g #2 Fri Jan 11 22:25:55 EST 2002 i686 unknown

Derived from:
htty://kernelnewbies.org/kernels/rh72/SOURCES/linux-2.4.2-vm-1-2-3-gbyte.patch Some parts of the patch above are already in the mainline trees.

Patch below applies to 2.4.18pre2aa2:

Rules.make Fri Jan 11 22:00:57 2002 renamed from Rules.make Tue Mar 6 22:31:01 2001 +17 -0
diff -nur linux.aa2/Rules.make linux/Rules.make
@@ -212,6 +212,7 @@
 #
 # Added the SMP separator to stop module accidents between uniprocessor
 # and SMP Intel boxes - AC - from bits by Michael Chastain
+# Added separator for different PAGE_OFFSET memory models - Ingo.
 #
 
 ifdef CONFIG_SMP
@@ -220,6 +221,22 @@
        genksyms_smp_prefix :=
 endif
 
+ifdef CONFIG_2GB
+ifdef CONFIG_SMP
+       genksyms_smp_prefix := -p smp_2gig_
+else
+       genksyms_smp_prefix := -p 2gig_
+endif
+endif
+
+ifdef CONFIG_3GB
+ifdef CONFIG_SMP
+       genksyms_smp_prefix := -p smp_3gig_
+else
+       genksyms_smp_prefix := -p 3gig_
+endif
+endif
+
 $(MODINCL)/%.ver: %.c
        @if [ ! -r $(MODINCL)/$*.stamp -o $(MODINCL)/$*.stamp -ot $< ]; then \
                echo '$(CC) $(CFLAGS) $(EXTRA_CFLAGS) -E -D__GENKSYMS__ $<'; \
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +6 -1
diff -nur linux.aa2/arch/i386/config.in linux/arch/i386/config.in
@@ -169,7 +169,11 @@
 if [ "$CONFIG_HIGHMEM64G" = "y" ]; then
    define_bool CONFIG_X86_PAE y
 else
-   bool '3.5GB user address space' CONFIG_05GB
+   choice 'Maximum Virtual Memory' \
+       "3GB            CONFIG_1GB \
+        2GB            CONFIG_2GB \
+        1GB            CONFIG_3GB \
+        05GB           CONFIG_05GB" 3GB
 fi
 if [ "$CONFIG_NOHIGHMEM" = "y" ]; then
    define_bool CONFIG_NO_PAGE_VIRTUAL y
@@ -179,6 +183,7 @@
    bool 'HIGHMEM I/O support (EXPERIMENTAL)' CONFIG_HIGHIO
 fi
 
+
 bool 'Math emulation' CONFIG_MATH_EMULATION
 bool 'MTRR (Memory Type Range Register) support' CONFIG_MTRR
 bool 'Symmetric multi-processing support' CONFIG_SMP
include/asm-i386/page_offset.h Fri Jan 11 21:20:48 2002 renamed from include/asm-i386/page_offset.h Fri Jan 11 20:57:58 2002 +7 -3
diff -nur linux.aa2/include/asm-i386/page_offset.h linux/include/asm-i386/page_offset.h
@@ -1,6 +1,10 @@
 #include <linux/config.h>
-#ifndef CONFIG_05GB
-#define PAGE_OFFSET_RAW 0xC0000000
-#else
+#ifdef CONFIG_05GB
 #define PAGE_OFFSET_RAW 0xE0000000
+#elif defined(CONFIG_1GB)
+#define PAGE_OFFSET_RAW 0xC0000000
+#elif defined(CONFIG_2GB)
+#define PAGE_OFFSET_RAW 0x80000000
+#elif defined(CONFIG_3GB)
+#define PAGE_OFFSET_RAW 0x40000000
 #endif
Signature · 1 line
Randy Hron
#1 lore raw
Followup to:  <[email protected]>
By author:    [email protected]
In newsgroup: linux.dev.kernel

arch/i386/config.in Fri Jan 11 22:20:32 2002 line 177

Quoted 13 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +5 -1
@@ -169,7 +169,11 @@
 if [ "$CONFIG_HIGHMEM64G" = "y" ]; then
    define_bool CONFIG_X86_PAE y
 else
-   bool '3.5GB user address space' CONFIG_05GB
+   choice 'Maximum Virtual Memory' \
+       "3GB            CONFIG_1GB \
+        2GB            CONFIG_2GB \
+        1GB            CONFIG_3GB \
+        05GB           CONFIG_05GB" 3GB
 fi

Calling this "Maximum Virtual Memory" is misleading at best. This is best described as "kernel:user split" (3:1, 2:2, 1:3, 3.5:0.5); "maximum virtual memory" sounds to me a lot like the opposite of what your parameter is.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#2 lore raw
In reply to [email protected]
Quoted 20 lines · On Sat, Jan 12, 2002 at 12:45:28AM -0500, [email protected] wrote:

Patch to have 1-3 GB of virtual memory and not show up as highmem:

Tested on uniprocessor Athlon with 1024 MB RAM and 1027 MB swap. Caused no LTP (ltp-20020108) regressions.
Did a test like http://marc.theaimsgroup.com/?l=linux-kernel&m=101064072924424&w=2
This time the test completed in 51 minutes (11% faster) and I had setiathome running the whole time and listened to 12 mp3s sampled at 128k.

dmesg|grep Mem
Memory: 1029848k/1048512k available (1054k kernel code, 18276k reserved, 260k data, 240k init, 0k highmem)

egrep '^CONFIG_HIGH|GB' /usr/src/linux/.config
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
# CONFIG_1GB is not set
CONFIG_2GB=y
# CONFIG_3GB is not set
# CONFIG_05GB is not set

Quoted 8 lines

uname -a
Linux rushmore 2.4.18pre2aa2-2g #2 Fri Jan 11 22:25:55 EST 2002 i686 unknown

Derived from:
htty://kernelnewbies.org/kernels/rh72/SOURCES/linux-2.4.2-vm-1-2-3-gbyte.patch Some parts of the patch above are already in the mainline trees.

Patch below applies to 2.4.18pre2aa2:

for a fileserver (even more if in kernel like tux) it certainly make sense to have as much direct mapped memory as possible, it is not the recommended setup for a generic purpose kernel though. So I applied the patch (except the prefix thing which is distribution specific). thanks,

Andrea

#3 lore raw
Quoted 5 lines · Andrea Arcangeli

for a fileserver (even more if in kernel like tux) it certainly make sense to have as much direct mapped memory as possible, it is not the recommended setup for a generic purpose kernel though. So I applied the patch (except the prefix thing which is distribution specific). thanks,

Please add in the patch below as well. It needs some explanation! A few weeks ago we noticed a compiler bug: in both egcs-2.91.66 and gcc-2.95.3; not in RH 2.96-85; forget if I tried gcc-3.0, expect okay.

If CONFIG_HIGHMEM64G (PAE: 3 levels of page table, 64-bit pte), the free_one_pgd code inlined in clear_page_tables is miscompiled:
the loop is terminated by a "jle" signed comparison of addresses instead of a "jb" unsigned comparison.

Usually not a problem: but if you configure for 1GB of user virtual and 3GB of kernel virtual, and you have more than 1GB of physical memory (as you normally would if chose HIGHMEM64G), then there's a page at physical address 0x3ffff000, directly mapped to virtual address 0x7ffff000. And if that page happens to get used for the pmd of a process, then on exit the free_one_pgd loop wraps over to carry on freeing "entries" at 0x80000000, 0x80000008, ... A lot of pmd_ERROR messages, but eventually an entry scrapes through the pmd_bad test and is wrongly freed, not so good.

The patch below seems to be enough to convince egcs-2.91.66 and gcc-2.95.3 to use a "jb" comparison there. I'm working on PIII, prefetchw() just a stub, if that makes any difference.

This patch is not actually what we've used. Paranoia (what other such bugs might there be?) drove me to set physical pages 0x3ffff and 0x40000 as Reserved in arch/i386/setup.c. I don't think it's appropriate to force that level of paranoia on others; but anyone configuring 3GBK should remember that it's a less-travelled path.

Hugh

mm/memory.c +4 -5
@@ -106,8 +106,7 @@
 
 static inline void free_one_pgd(pgd_t * dir)
 {
-	int j;
-	pmd_t * pmd;
+	pmd_t * pmd, * md, * emd;
 
 	if (pgd_none(*dir))
 		return;
@@ -118,9 +117,9 @@
 	}
 	pmd = pmd_offset(dir, 0);
 	pgd_clear(dir);
-	for (j = 0; j < PTRS_PER_PMD ; j++) {
-		prefetchw(pmd+j+(PREFETCH_STRIDE/16));
-		free_one_pmd(pmd+j);
+	for (md = pmd, emd = pmd + PTRS_PER_PMD; md < emd; md++) {
+		prefetchw(md+(PREFETCH_STRIDE/16));
+		free_one_pmd(md);
 	}
 	pmd_free(pmd);
 }
#4 lore raw
In reply to Andrea Arcangeli

Review

Quoted 10 lines · Hugh Dickins

...
The patch below seems to be enough to convince egcs-2.91.66 and gcc-2.95.3 to use a "jb" comparison there.
...

(quoted diff) +3 -3
-       for (j = 0; j < PTRS_PER_PMD ; j++) {
-               prefetchw(pmd+j+(PREFETCH_STRIDE/16));
-               free_one_pmd(pmd+j);
+       for (md = pmd, emd = pmd + PTRS_PER_PMD; md < emd; md++) {
+               prefetchw(md+(PREFETCH_STRIDE/16));
+               free_one_pmd(md);

You need to add a big fat comment here, describing the compiler problem, and the risks which attend any change to this code.

-

#5 lore raw
In reply to Andrea Arcangeli
Followup to:  <[email protected]>
By author:    Hugh Dickins <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 11 lines

Usually not a problem: but if you configure for 1GB of user virtual and 3GB of kernel virtual, and you have more than 1GB of physical memory (as you normally would if chose HIGHMEM64G), then there's a page at physical address 0x3ffff000, directly mapped to virtual address 0x7ffff000. And if that page happens to get used for the pmd of a process, then on exit the free_one_pgd loop wraps over to carry on freeing "entries" at 0x80000000, 0x80000008, ... A lot of pmd_ERROR messages, but eventually an entry scrapes through the pmd_bad test and is wrongly freed, not so good.

By the way, expect user programs to fail due to lack of address space if you only give them 1 GB of userspace. At 1 GB of userspace there is *no* address space which is compatible with the normal address space map available to the user process.

I would personally vote against including that particular option.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#6 lore raw
Quoted 6 lines · H. Peter Anvin

By the way, expect user programs to fail due to lack of address space if you only give them 1 GB of userspace. At 1 GB of userspace there is *no* address space which is compatible with the normal address space map available to the user process.

I would personally vote against including that particular option.

It could be useful for machines where most activity happens inside the kernel, though. Think of TUX web or ftp servers or dedicated NFS servers.

regards,

Rik

Signature · 4 lines
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/
#7 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · H. Peter Anvin

By the way, expect user programs to fail due to lack of address space if you only give them 1 GB of userspace. At 1 GB of userspace there is *no* address space which is compatible with the normal address space map available to the user process.

Actually, I think it will work for apps < `600MB since the mmap area is automatically adjusted to begin at PAGE_OFFSET/3.

cat /proc/1/maps

CONFIG_1GB
08048000-0804e000 r-xp 00000000 03:41 58716 /sbin/init
0804e000-08050000 rw-p 00005000 03:41 58716 /sbin/init
08050000-08054000 rwxp 00000000 00:00 0
40000000-40016000 r-xp 00000000 03:41 73822 /lib/ld-2.2.4.so
40016000-40017000 rw-p 00015000 03:41 73822 /lib/ld-2.2.4.so
40017000-40018000 rw-p 00000000 00:00 0
4002c000-4015e000 r-xp 00000000 03:41 73816 /lib/i686/libc-2.2.4.so
4015e000-40164000 rw-p 00131000 03:41 73816 /lib/i686/libc-2.2.4.so
40164000-40168000 rw-p 00000000 00:00 0
bfffe000-c0000000 rwxp fffff000 00:00 0

CONFIG_3GB
08048000-0804e000 r-xp 00000000 03:41 58716 /sbin/init
0804e000-08050000 rw-p 00005000 03:41 58716 /sbin/init
08050000-08054000 rwxp 00000000 00:00 0
15556000-1556c000 r-xp 00000000 03:41 73822 /lib/ld-2.2.4.so
1556c000-1556d000 rw-p 00015000 03:41 73822 /lib/ld-2.2.4.so
1556d000-1556e000 rw-p 00000000 00:00 0
15582000-156b4000 r-xp 00000000 03:41 73816 /lib/i686/libc-2.2.4.so
156b4000-156ba000 rw-p 00131000 03:41 73816 /lib/i686/libc-2.2.4.so
156ba000-156be000 rw-p 00000000 00:00 0
3fffe000-40000000 rwxp fffff000 00:00 0

Then again, I agree the 3GB option doesn't make much sense for 99.99% of cases.

-Marvin

#8 lore raw
In reply to Andrea Arcangeli
Quoted 9 lines · Marvin Justice
Quoted 5 lines · H. Peter Anvin

By the way, expect user programs to fail due to lack of address space if you only give them 1 GB of userspace. At 1 GB of userspace there is *no* address space which is compatible with the normal address space map available to the user process.

Actually, I think it will work for apps < `600MB since the mmap area is automatically adjusted to begin at PAGE_OFFSET/3.

As I said: At 1 GB of userspace there is *no* address space which is compatible with the normal address space map available to the user process.

There is mmap() space, available, sure, but you can't get the same address, even by request. Applications that care about the layout of the address space will fail.

-hpa

#9 lore raw
Quoted 6 lines

As I said: At 1 GB of userspace there is *no* address space which is compatible with the normal address space map available to the user process.

There is mmap() space, available, sure, but you can't get the same address, even by request. Applications that care about the layout of the address space will fail.

That sounds a good reason to run this mode a bit for debugging (be sure to use Hugh's gcc 2.95/egcs-1.1.2 bug fix when trying this though!)

#10 lore raw
In reply to Andrea Arcangeli

Hi!

Quoted 3 lines

The patch below seems to be enough to convince egcs-2.91.66 and gcc-2.95.3 to use a "jb" comparison there. I'm working on PIII, prefetchw() just a stub, if that makes any difference.

If this is really gcc bug, would simply making j volatile fix it?

Pavel

Quoted 25 lines
mm/memory.c +4 -5
@@ -106,8 +106,7 @@
 
 static inline void free_one_pgd(pgd_t * dir)
 {
-	int j;
-	pmd_t * pmd;
+	pmd_t * pmd, * md, * emd;
 
 	if (pgd_none(*dir))
 		return;
@@ -118,9 +117,9 @@
 	}
 	pmd = pmd_offset(dir, 0);
 	pgd_clear(dir);
-	for (j = 0; j < PTRS_PER_PMD ; j++) {
-		prefetchw(pmd+j+(PREFETCH_STRIDE/16));
-		free_one_pmd(pmd+j);
+	for (md = pmd, emd = pmd + PTRS_PER_PMD; md < emd; md++) {
+		prefetchw(md+(PREFETCH_STRIDE/16));
+		free_one_pmd(md);
 	}
 	pmd_free(pmd);
 }
Signature · 2 lines
(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa
#11 lore raw
Quoted 6 lines · Pavel Machek
Quoted 4 lines

The patch below seems to be enough to convince egcs-2.91.66 and gcc-2.95.3 to use a "jb" comparison there. I'm working on PIII, prefetchw() just a stub, if that makes any difference.

If this is really gcc bug, would simply making j volatile fix it?

You rogue! The panacea, eh? Well, yes, it does look like that's enough with egcs-2.91.66 (I don't have 2.95 here to try at the moment, expect it would behave the same) - the comparison uses "jle" as before, but now it's correctly on free_one_pmd's index j instead of an address. Neat - but an even bigger fatter comment needed?

Hugh

#12 lore raw
In reply to [email protected]
Quoted 13 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    [email protected]
In newsgroup: linux.dev.kernel
Quoted 10 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +3 -1
@@ -169,7 +169,11 @@
 if [ "$CONFIG_HIGHMEM64G" = "y" ]; then
    define_bool CONFIG_X86_PAE y
 else
-   bool '3.5GB user address space' CONFIG_05GB
+   choice 'Maximum Virtual Memory' \
+       "3GB            CONFIG_1GB \
+        2GB            CONFIG_2GB \
Quoted 8 lines

arch/i386/config.in Fri Jan 11 22:20:32 2002 line 177

Quoted 3 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +2 -0
+        1GB            CONFIG_3GB \
+        05GB           CONFIG_05GB" 3GB
 fi

Calling this "Maximum Virtual Memory" is misleading at best. This is best described as "kernel:user split" (3:1, 2:2, 1:3, 3.5:0.5); "maximum virtual memory" sounds to me a lot like the opposite of what your parameter is.

actually it is really max virtual memory.. but from the user point of view, user is supposed to care about the virtual memory he can manage, not about what the kernel will do with the rest. So if the user wants 3GB of virtual memory available to each task he will select 3GB. I really don't mind if you want to change it from the kernel point of view, but given it's the user who's supposed to compile it, also the current patch looks good enough to me.

Andrea

#13 lore raw
Quoted 22 lines · Andrea Arcangeli
Quoted 12 lines · H. Peter Anvin

By author: [email protected]

Quoted 10 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +3 -1
@@ -169,7 +169,11 @@
 if [ "$CONFIG_HIGHMEM64G" = "y" ]; then
    define_bool CONFIG_X86_PAE y
 else
-   bool '3.5GB user address space' CONFIG_05GB
+   choice 'Maximum Virtual Memory' \
+       "3GB            CONFIG_1GB \
+        2GB            CONFIG_2GB \
Quoted 8 lines

arch/i386/config.in Fri Jan 11 22:20:32 2002 line 177

Quoted 3 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +2 -0
+        1GB            CONFIG_3GB \
+        05GB           CONFIG_05GB" 3GB
 fi

Calling this "Maximum Virtual Memory" is misleading at best. This is best described as "kernel:user split" (3:1, 2:2, 1:3, 3.5:0.5); "maximum virtual memory" sounds to me a lot like the opposite of what your parameter is.

Quoted 7 lines

actually it is really max virtual memory.. but from the user point of view, user is supposed to care about the virtual memory he can manage, not about what the kernel will do with the rest. So if the user wants 3GB of virtual memory available to each task he will select 3GB. I really don't mind if you want to change it from the kernel point of view, but given it's the user who's supposed to compile it, also the current patch looks good enough to me.

The numbers are wrong anyway, because of vmalloc() and PCI space. The PCI space is motherboard-dependent AFAIK, but you could at least account for the 128 MB vmalloc() area:

user virtual space / non-kmap physical memory

3584/384
3072/896
2048/1920
1024/2944 (sure this works, even for syscalls w/ bad pointers?)
512/3456 (sure this works, even for syscalls w/ bad pointers?)

#14 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · Albert D. Cahalan
Quoted 2 lines · Andrea Arcangeli
Quoted 1 line · H. Peter Anvin

By author: [email protected]

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +0 -0
@@ -169,7 +169,11 @@
 if [ "$CONFIG_HIGHMEM64G" = "y" ]; then
Quoted 8 lines
Quoted 8 lines

arch/i386/config.in Fri Jan 11 22:20:32 2002 line 176

Quoted 8 lines
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +5 -1
    define_bool CONFIG_X86_PAE y
 else
-   bool '3.5GB user address space' CONFIG_05GB
+   choice 'Maximum Virtual Memory' \
+       "3GB            CONFIG_1GB \
+        2GB            CONFIG_2GB \
+        1GB            CONFIG_3GB \
+        05GB           CONFIG_05GB" 3GB

^^ this should be 3.5GB btw

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines

arch/i386/config.in Fri Jan 11 22:20:32 2002 line 177

Quoted 1 line
arch/i386/config.in Fri Jan 11 22:20:32 2002 renamed from arch/i386/config.in Fri Jan 11 20:57:58 2002 +0 -0
 fi

Calling this "Maximum Virtual Memory" is misleading at best. This is best described as "kernel:user split" (3:1, 2:2, 1:3, 3.5:0.5); "maximum virtual memory" sounds to me a lot like the opposite of what your parameter is.

Quoted 7 lines

actually it is really max virtual memory.. but from the user point of view, user is supposed to care about the virtual memory he can manage, not about what the kernel will do with the rest. So if the user wants 3GB of virtual memory available to each task he will select 3GB. I really don't mind if you want to change it from the kernel point of view, but given it's the user who's supposed to compile it, also the current patch looks good enough to me.

Quoted 3 lines

The numbers are wrong anyway, because of vmalloc() and PCI space. The PCI space is motherboard-dependent AFAIK, but you could at least account for the 128 MB vmalloc() area:

looks dirty, the size of the kernel direct mapping is mainly in function of #defines that can be changed freely, they're not constant in function of CONFIG_1G etc.. and it changes also in function of smp/up/4G/64G options. The 3GB/2GB/1GB/3.5GB visible into the menuconfig are exact instead. So I wouldn't mention inprecise stuff that can changed under us (and the exact size of the kernel direct mapping doesn't matter to the user anyways I think [and if it matters I think it means he's skilled enough to know about vmalloc space ;) ]).

Quoted 8 lines

user virtual space / non-kmap physical memory

3584/384
3072/896
2048/1920
1024/2944 (sure this works, even for syscalls w/ bad pointers?)
512/3456 (sure this works, even for syscalls w/ bad pointers?)

Andrea

#15 lore raw
Quoted 6 lines · Andrea Arcangeli
Quoted 3 lines · Albert D. Cahalan

The numbers are wrong anyway, because of vmalloc() and PCI space. The PCI space is motherboard-dependent AFAIK, but you could at least account for the 128 MB vmalloc() area:

Quoted 8 lines

looks dirty, the size of the kernel direct mapping is mainly in function of #defines that can be changed freely, they're not constant in function of CONFIG_1G etc.. and it changes also in function of smp/up/4G/64G options. The 3GB/2GB/1GB/3.5GB visible into the menuconfig are exact instead. So I wouldn't mention inprecise stuff that can changed under us (and the exact size of the kernel direct mapping doesn't matter to the user anyways I think [and if it matters I think it means he's skilled enough to know about vmalloc space ;) ]).

The problem is that the "1GB" option doesn't cover 1 GB. It is common for people to buy 1 GB of memory (power of 2), and then complain that Linux only sees 896 MB of memory.

So how will you make the choices clear to such people? There are 3 options, not counting the slram block device:

a. give up 128 MB (12.5 %) of memory
b. suffer the kmap overhead
c. give up some user virtual address space

None of this is obvious. The user sees "1GB" and will innocently believe that this is good for a 1 GB system!

BTW, do we no longer require that the kernel side of things (whole thing, including vmalloc space) be a power of two? There used to be a bitwise operation in the user access stuff.

#16 lore raw

BIO Usage Error or Conflicting Designs

Jens,

Below is a single sector read using ACB.
If I do not use the code inside "#ifdef USEBIO" and run UP/SMP but no highmem, it runs and works like a charm. It is also 100% unchanged code from what is in 2.4 patches. The attached oops is generate under SMP without highmem and running the USEBIO code.

CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set

Regards,

Andre Hedrick
Linux Disk Certification Project Linux ATA Development

/*
* Handler for command with PIO data-in phase
*/
ide_startstop_t task_in_intr (ide_drive_t *drive)
{

  byte stat               = GET_STAT();
  byte io_32bit           = drive->io_32bit;
  struct request *rq      = HWGROUP(drive)->rq;
  char *pBuf              = NULL;
  if (!OK_STAT(stat,DATA_READY,BAD_R_STAT)) {
    if (stat & (ERR_STAT|DRQ_STAT)) {
      return ide_error(drive, "task_in_intr", stat);
    }
    if (!(stat & BUSY_STAT)) {
      DTF("task_in_intr to Soon wait for next interrupt\n");
      ide_set_handler(drive, &task_in_intr, WAIT_CMD, NULL);
      return ide_started;
    }

}
drive->io_32bit = 0;
DTF("stat: %02x\n", stat);
#ifdef USEBIO

  if (rq->flags & REQ_CMD) {
    pBuf = ide_map_buffer(rq, &flags);
  } else {
    pBuf = rq->buffer + ((rq->nr_sectors - rq->current_nr_sectors) * SECTOR_SIZE);

}
#else
pBuf = rq->buffer + ((rq->nr_sectors - rq->current_nr_sectors) * SECTOR_SIZE);
#endif
DTF("Read: %p, rq->current_nr_sectors: %d\n", pBuf, (int) rq->current_nr_sectors); taskfile_input_data(drive, pBuf, SECTOR_WORDS);
#ifdef USEBIO
if (rq->flags & REQ_CMD)

    ide_unmap_buffer(pBuf, &flags);

rq->sector++;
rq->errors = 0;
#endif
drive->io_32bit = io_32bit;

  if (--rq->current_nr_sectors <= 0) {
    /* (hs): swapped next 2 lines */
    DTF("Request Ended stat: %02x\n", GET_STAT());
    ide_end_request(1, HWGROUP(drive));
  } else {
    ide_set_handler(drive, &task_in_intr,  WAIT_CMD, NULL);
    return ide_started;

}
return ide_stopped;
}

ksymoops 2.3.5 on i686 2.5.2-pre11. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.5.2-pre11/ (default)
-m /usr/src/linux/System.map (specified)

No modules in ksyms, skipping objects

Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid lsmod file?
cpu: 0, clocks: 2659849, slice: 886616
cpu: 1, clocks: 2659849, slice: 886616
invalid operand: 0000
CPU:    0
EIP:    0010:[<c0136dd9>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010283
eax: 00038000 ebx: 00000070 ecx: f70e4980 edx: f70e4980
esi: c0402084 edi: 00038000 ebp: f70e4980 esp: f6a01df4
ds: 0018 es: 0018 ss: 0018
Process hdparm (pid: 822, stackpage=f6a01000)

Stack: 00000000 c0402084 00000002 f70e4980 00001000 00000001 00000000 00000217 
       00000000 00000000 00000000 00000000 c01fcafe 00038000 00000070 f6a01e68 
       00000000 f7d10640 c0402084 f70e4980 00000002 00000000 00000000 00000002 
Call Trace: [<c01fcafe>] [<c01fd036>] [<c01fd14c>] [<c01fd1f9>] [<c013be3b>] 
   [<c013f00f>] [<c013ef70>] [<c012b7ce>] [<c012bc82>] [<c012bb90>] [<c01390ce>] 
   [<c0108c7f>] 
Code: 0f 0b c7 44 24 24 70 00 00 00 8b 1d 68 08 3e c0 89 5c 24 20 
Quoted 1 line
Quoted 1 line

EIP; c0136dd9 <create_bounce+49/2c0> <=====

Trace; c01fcafe <__make_request+5e/470>
Trace; c01fd036 <generic_make_request+126/1b0>
Trace; c01fd14c <submit_bio+4c/60>
Trace; c01fd1f9 <submit_bh+99/a0>
Trace; c013be3b <block_read_full_page+1db/1f0>
Trace; c013f00f <blkdev_readpage+f/20>
Trace; c013ef70 <blkdev_get_block+0/40>
Trace; c012b7ce <do_generic_file_read+2de/460>
Trace; c012bc82 <generic_file_read+92/190>
Trace; c012bb90 <file_read_actor+0/60>
Trace; c01390ce <sys_read+8e/d0>
Trace; c0108c7f <system_call+33/38>
Code; c0136dd9 <create_bounce+49/2c0>
00000000 <_EIP>:
Code; c0136dd9 <create_bounce+49/2c0> <=====
0: 0f 0b ud2a <=====
Code; c0136ddb <create_bounce+4b/2c0>
2: c7 44 24 24 70 00 00 movl $0x70,0x24(%esp,1)
Code; c0136de2 <create_bounce+52/2c0>
9: 00
Code; c0136de3 <create_bounce+53/2c0>
a: 8b 1d 68 08 3e c0 mov 0xc03e0868,%ebx
Code; c0136de9 <create_bounce+59/2c0>
10: 89 5c 24 20 mov %ebx,0x20(%esp,1)

1 warning issued. Results may not be reliable.

#17 lore raw

Re: BIO Usage Error or Conflicting Designs

In reply to Albert D. Cahalan
Quoted 10 lines · Andre Hedrick

Jens,

Below is a single sector read using ACB.
If I do not use the code inside "#ifdef USEBIO" and run UP/SMP but no highmem, it runs and works like a charm. It is also 100% unchanged code from what is in 2.4 patches. The attached oops is generate under SMP without highmem and running the USEBIO code.

CONFIG_NOHIGHMEM=y

Is this with the highmem debug stuff enabled? That's the only way I can see this BUG triggering, otherwise q->bounce_pfn _cannot_ be smaller than the max_pfn.

Signature · 1 line
Jens Axboe
#18 lore raw

Re: BIO Usage Error or Conflicting Designs

Jens,

Here is back at you sir.

Andre Hedrick
Linux Disk Certification Project Linux ATA Development

ksymoops 2.3.5 on i686 2.5.2-pre11. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.5.2-pre11/ (default)
-m /usr/src/linux/System.map (specified)

No modules in ksyms, skipping objects
Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid lsmod file?
cpu: 0, clocks: 2660067, slice: 886689
cpu: 1, clocks: 2660067, slice: 886689
kernel BUG at highmem.c:374!

invalid operand: 0000
CPU:    1
EIP:    0010:[<c0144478>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286
eax: 0000001d ebx: 00000070 ecx: c03785ac edx: 00003ce8
esi: c045ea64 edi: 00038000 ebp: f76bf2c4 esp: f6321de4
ds: 0018 es: 0018 ss: 0018
Process hdparm (pid: 924, stackpage=f6321000)

Stack: c02fe555 00000176 00000000 c045ea64 00000002 f76bf2c4 00000001 00000246 
       c1effecc 00000000 f1aabd84 00000020 00000000 00000000 c022092e 00038000 
       00000070 f6321e60 00000000 f7c33344 c045ea64 f76bf2c4 00000002 00000000 
Call Trace: [<c022092e>] [<c020f21e>] [<c0220f9f>] [<c02210f1>] [<c02211d9>] 
   [<c014b31c>] [<c014f46f>] [<c014f3d0>] [<c013477e>] [<c0134cc2>] [<c0134bd0>] 
   [<c014706e>] [<c010935f>] 
Code: 0f 0b 83 c4 08 c7 44 24 24 70 00 00 00 8b 1d 68 b9 43 c0 8b 
Quoted 1 line
Quoted 1 line

EIP; c0144478 <create_bounce+68/310> <=====

Trace; c022092e <__make_request+5e/590>
Trace; c020f21e <set_cursor+6e/80>
Trace; c0220f9f <generic_make_request+13f/1c0>
Trace; c02210f1 <submit_bio+81/90>
Trace; c02211d9 <submit_bh+d9/e0>
Trace; c014b31c <block_read_full_page+1ec/200>
Trace; c014f46f <blkdev_readpage+f/20>
Trace; c014f3d0 <blkdev_get_block+0/40>
Trace; c013477e <do_generic_file_read+34e/560>
Trace; c0134cc2 <generic_file_read+92/190>
Trace; c0134bd0 <file_read_actor+0/60>
Trace; c014706e <sys_read+8e/d0>
Trace; c010935f <system_call+33/38>
Code; c0144478 <create_bounce+68/310>
00000000 <_EIP>:
Code; c0144478 <create_bounce+68/310> <=====
0: 0f 0b ud2a <=====
Code; c014447a <create_bounce+6a/310>
2: 83 c4 08 add $0x8,%esp
Code; c014447d <create_bounce+6d/310>
5: c7 44 24 24 70 00 00 movl $0x70,0x24(%esp,1)
Code; c0144484 <create_bounce+74/310>
c: 00
Code; c0144485 <create_bounce+75/310>
d: 8b 1d 68 b9 43 c0 mov 0xc043b968,%ebx
Code; c014448b <create_bounce+7b/310>
13: 8b 00 mov (%eax),%eax

1 warning issued. Results may not be reliable.

[root@xathy DiskPerf-1.0.7]# hdparm -t /dev/hde

/dev/hde:
Timing buffered disk reads: kernel BUG at highmem.c:374!

invalid operand: 0000
CPU:    1
EIP:    0010:[<c0144478>]    Not tainted
EFLAGS: 00010286
eax: 0000001d   ebx: 00000070   ecx: c03785ac   edx: 00003ce8
esi: c045ea64   edi: 00038000   ebp: f76bf2c4   esp: f6321de4
ds: 0018   es: 0018   ss: 0018
Process hdparm (pid: 924, stackpage=f6321000)
Stack: c02fe555 00000176 00000000 c045ea64 00000002 f76bf2c4 00000001 00000246
       c1effecc 00000000 f1aabd84 00000020 00000000 00000000 c022092e 00038000
       00000070 f6321e60 00000000 f7c33344 c045ea64 f76bf2c4 00000002 00000000
Call Trace: [<c022092e>] [<c020f21e>] [<c0220f9f>] [<c02210f1>] [<c02211d9>]
   [<c014b31c>] [<c014f46f>] [<c014f3d0>] [<c013477e>] [<c0134cc2>] [<c0134bd0>]
   [<c014706e>] [<c010935f>]

Code: 0f 0b 83 c4 08 c7 44 24 24 70 00 00 00 8b 1d 68 b9 43 c0 8b
 Segmentation fault

[root@xathy DiskPerf-1.0.7]# hdparm -v /dev/hde

/dev/hde:

 multcount    =  0 (off)
 I/O support  =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  0 (off)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)

geometry = 55488/255/63, sectors = 320173056, start = 0

#19 lore raw

Re: BIO Usage Error or Conflicting Designs

In reply to Jens Axboe
Quoted 4 lines · Andre Hedrick

Jens,

Here is back at you sir.

Without highmem debug enabled?? I already knew this was the bug triggered, nothing new here.

Please print the two pfn values triggering the BUG_ON, I'll take a look at this tomorrow.

Signature · 1 line
Jens Axboe
#20 lore raw

Re: BIO Usage Error or Conflicting Designs

Quoted 11 lines · Jens Axboe
Quoted 4 lines · Andre Hedrick

Jens,

Here is back at you sir.

Without highmem debug enabled?? I already knew this was the bug triggered, nothing new here.

Please print the two pfn values triggering the BUG_ON, I'll take a look at this tomorrow.

That is with highmem debug on, the stuff at the end of the config file. Nothing more is generated, if there are more flags to set please tell me where.

Regards,

Andre Hedrick
Linux Disk Certification Project Linux ATA Development

#21 lore raw

Re: BIO Usage Error or Conflicting Designs

In reply to Jens Axboe
Quoted 11 lines · Andre Hedrick
Quoted 9 lines · Jens Axboe
Quoted 4 lines · Andre Hedrick

Jens,

Here is back at you sir.

Without highmem debug enabled?? I already knew this was the bug triggered, nothing new here.

Quoted 6 lines
Quoted 2 lines

Please print the two pfn values triggering the BUG_ON, I'll take a look at this tomorrow.

That is with highmem debug on, the stuff at the end of the config file. Nothing more is generated, if there are more flags to set please tell me where.

Sorry if I wasn't clear, I mean the emulate highmem debug patch I forwarded to you. I'll look into Manfred's post right now, you can simply remove the

#ifndef CONFIG_HIGHMEM
	BUG();

#endif

test for now, for testing.

Signature · 1 line
Jens Axboe
#22 lore raw
In reply to [email protected]
Quoted 4 lines
Quoted 4 lines

Derived from:
htty://kernelnewbies.org/kernels/rh72/SOURCES/linux-2.4.2-vm-1-2-3-gbyte.patch Some parts of the patch above are already in the mainline trees.

Quoted 8 lines
Quoted 1 line

Patch below applies to 2.4.18pre2aa2:

for a fileserver (even more if in kernel like tux) it certainly make sense to have as much direct mapped memory as possible, it is not the recommended setup for a generic purpose kernel though. So I applied the patch (except the prefix thing which is distribution specific). thanks,

Andrea

Thanks so much!

Signature · 1 line
Randy Hron
#23 lore raw
In reply to [email protected]
Quoted 9 lines · Andrea Arcangeli

actually it is really max virtual memory.. but from the user point of view, user is supposed to care about the virtual memory he can manage, not about what the kernel will do with the rest. So if the user wants 3GB of virtual memory available to each task he will select 3GB. I really don't mind if you want to change it from the kernel point of view, but given it's the user who's supposed to compile it, also the current patch looks good enough to me.

Oh, right... if so he simply has the 05G option mislabelled... it should be 3.5G

-hpa

#24 lore raw
In reply to [email protected]

Based on some of the comments in the thread, here is what I came up with for Configure.help.

Also, in the patch, I had 3GB as the default config option. It may be safer to have 1GB as the default configure option to match the mainline.

Documentation/Configure.help Sat Jan 12 16:29:21 2002 renamed from Documentation/Configure.help Fri Jan 11 20:57:58 2002 +53 -0
@@ -376,6 +376,59 @@
   Select this if you have a 32-bit processor and more than 4
   gigabytes of physical RAM.
 
+# Choice: maxvm
+Maximum Virtual Memory
+CONFIG_1GB
+  If you have 4 Gigabytes of physical memory or less, you can change
+  where the where the kernel maps high memory.  If you have less
+  than 1 gigabyte of physical memory, you should disable
+  CONFIG_HIGHMEM4G because you don't need the choices below.
+
+  If you have a large amount of physical memory, all of it may not
+  be "permanently mapped" by the kernel. The physical memory that
+  is not permanently mapped is called "high memory".
+
+  The numbers in the configuration options are not precise because
+  of the kernel's vmalloc() area, and the PCI space on motherboards
+  may vary as well.  Typically there will 128 megabytes less
+  "user memory" mapped than the number in the configuration option.
+  Saying that another way, "high memory" will usually start 128
+  megabytes lower than the configuration option.
+
+  Selecting "05GB" results in a "3.5GB/0.5GB" kernel/user split:
+  3.5 gigabytes are kernel mapped so each process sees a 3.5
+  gigabyte virtual memory space and the remaining part of the 4
+  gigabyte virtual memory space is used by the kernel to permanently
+  map as much physical memory as possible.  On a system with 1 gigabyte
+  of physical memory, you may get 384 megabytes of "user memory" and
+  640 megabytes of "high memory" with this selection.
+
+  Selecting "1GB" results in a "3GB/1GB" kernel/user split:
+  3 gigabytes are mapped so each process sees a 3 gigabyte virtual
+  memory space and the remaining part of the 4 gigabyte virtual memory
+  space is used by the kernel to permanently map as much physical
+  memory as possible.  On a system with 1 gigabyte of memory, you may
+  get 896 MB of "user memory" and 128 megabytes of "high memory"
+
+  Selecting "2GB" results in a "2GB/2GB" kernel/user split:
+  2 gigabytes are mapped so each process sees a 2 gigabyte virtual
+  memory space and the remaining part of the 4 gigabyte virtual memory
+  space is used by the kernel to permanently map as much physical
+  memory as possible.  On a system with 1 to 1.75 gigabytes of
+  physical memory, this option have all make it so no memory is
+  mapped as "high memory".
+
+  Selecting "3GB" results in a "1GB/3GB" kernel/user split:
+  1 gigabyte is mapped so each process sees a 1 gigabyte virtual
+  memory space and the remaining part of the 4 gigabytes of virtual
+  memory space is used by the kernel to permanently map as much
+  physical memory as possible.
+
+  Options "2GB" and "3GB" may expose bugs that were dormant in
+  certain hardware and possibly even the kernel.
+
+  If unsure, say "1GB".
+
 HIGHMEM I/O support
 CONFIG_HIGHIO
   If you want to be able to do I/O to high memory pages, say Y.
Signature · 1 line
Randy Hron
#25 lore raw
In reply to [email protected]

This configure help may be clearer than the previous.

Documentation/Configure.help Sat Jan 12 17:27:01 2002 renamed from Documentation/Configure.help Fri Jan 11 20:57:58 2002 +44 -0
@@ -376,6 +376,50 @@
   Select this if you have a 32-bit processor and more than 4
   gigabytes of physical RAM.
 
+# Choice: maxvm
+Maximum Virtual Memory
+CONFIG_1GB
+  If you have 4 Gigabytes of physical memory or less, you can change
+  where the kernel maps high memory.  If you have less than 1 gigabyte
+  of physical memory, you should disable CONFIG_HIGHMEM4G because you
+  don't need the choices below.
+
+  If you have a large amount of physical memory, all of it may not
+  be "permanently mapped" by the kernel. The physical memory that
+  is not permanently mapped is called "high memory".
+
+  The numbers in the configuration options are not precise because
+  of the kernel's vmalloc() area, and the PCI space on motherboards
+  may vary as well.  Typically there will 128 megabytes less
+  "user memory" mapped than the number in the configuration option.
+  Saying that another way, "high memory" will usually start 128
+  megabytes lower than the configuration option.
+
+  Selecting "05GB" results in a "3.5GB/0.5GB" kernel/user split:
+  3.5 gigabytes are kernel mapped so each process sees a 3.5
+  gigabyte virtual memory space and the remaining part of the 4
+  gigabyte virtual memory space is used by the kernel to permanently
+  map as much physical memory as possible.  On a system with 1 gigabyte
+  of physical memory, you may get 384 megabytes of "user memory" and
+  640 megabytes of "high memory" with this selection.
+
+  Selecting "1GB" results in a "3GB/1GB" kernel/user split:
+  On a system with 1 gigabyte of memory, you may get 896 megabytes
+  of "user memory" and 128 megabytes of "high memory"
+
+  Selecting "2GB" results in a "2GB/2GB" kernel/user split:
+  On a system with 1 to 1.75 gigabytes of physical memory, this
+  option will make it so no memory is mapped as "high memory".
+
+  Selecting "3GB" results in a "1GB/3GB" kernel/user split:
+  On a system with 1 to 2.75 gigabytes of physical memory, this
+  option will make it so no memory is mapped as "high memory".
+
+  Options "2GB" and "3GB" may expose bugs that were dormant in
+  certain hardware and possibly even the kernel.
+
+  If unsure, say "1GB".
+
 HIGHMEM I/O support
 CONFIG_HIGHIO
   If you want to be able to do I/O to high memory pages, say Y.
Signature · 1 line
Randy Hron