Skip to contentSkip to conversation

2.6.26-rc1 lost half the RAM on UltraSPARC 5

BUG Mikael Pettersson 3 participants 16 messages May 7, 2008 – Sep 9, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it reports having only 128MB RAM, while earlier kernels reported the correct amount: 256MB.

A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

dmesg-2.6.26-rc1 renamed from dmesg-2.6.25 +25 -13
@@ -1,24 +1,36 @@
 PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
 PROMLIB: Root node compatible: 
-Linux version 2.6.25 (mikpe@sparge) (gcc version 4.2.3) #1 Thu Apr 17 19:53:16 CEST 2008
+Linux version 2.6.26-rc1 (mikpe@sparge) (gcc version 4.2.3) #1 Wed May 7 17:41:33 CEST 2008
 console [earlyprom0] enabled
 ARCH: SUN4U
 Ethernet address: 08:00:20:fd:ec:1f
 Kernel: Using 1 locked TLB entries for main kernel image.
 Remapping the kernel... done.
-[0000000200000000-fffff80000400000] page_structs&2144 node=0 entry=0/0
-[0000000200000000-fffff80000800000] page_structs&2144 node=0 entry=1/0
-[0000000200000000-fffff80000c00000] page_structs&2144 node=0 entry=2/0
-[0000000200000000-fffff80001000000] page_structs&2144 node=0 entry=3/0
 OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
-PROM: Built device tree with 46848 bytes of memory.
-On node 0 totalpages: 32298
+PROM: Built device tree with 46641 bytes of memory.
+Top of RAM: 0x17f46000, Total RAM: 0xff40000
+Memory hole size: 128MB
+Entering add_active_range(0, 0, 16384) 0 entries of 256 used
+Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
+Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
+Entering add_active_range(0, 49055, 49059) 3 entries of 256 used
+[0000000200000000-fffff80000400000] page_structs1072 node=0 entry=0/0
+[0000000200000000-fffff80000800000] page_structs1072 node=0 entry=1/0
+Zone PFN ranges:
+  Normal          0 ->    49059
+Movable zone start PFN for each node
+early_node_map[4] active PFN ranges
+    0:        0 ->    16384
+    0:    32768 ->    49023
+    0:    49024 ->    49053
+    0:    49055 ->    49059
+On node 0 totalpages: 32672
   Normal zone: 335 pages used for memmap
   Normal zone: 0 pages reserved
-  Normal zone: 31963 pages, LIFO batch:7
+  Normal zone: 32337 pages, LIFO batch:7
   Movable zone: 0 pages used for memmap
 Booting Linux...
-Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 31963
+Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32337
 Kernel command line: ro root=/dev/sda5
 PID hash table entries: 1024 (order: 10, 8192 bytes)
 clocksource: mult[28000] shift[16]
@@ -27,11 +39,11 @@
 console handover: boot [earlyprom0] -> real [tty0]
 Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
 Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
-Memory: 237256k available (1928k kernel code, 744k data, 144k init) [fffff80000000000,0000000017f46000]
-SLUB: Genslabs, HWalign2, Order=0-2, MinObjects=8, CPUs=1, Nodes=1
-Calibrating delay using timer specific routine.. 800.81 BogoMIPS (lpj@04065)
+Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]
+SLUB: Genslabs, HWalign2, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
+Calibrating delay using timer specific routine.. 800.99 BogoMIPS (lpj@04959)
 Mount-cache hash table entries: 512
-net_namespace: 280 bytes
+net_namespace: 344 bytes
 NET: Registered protocol family 16
 PCI: Probing for controllers.
 /pci@1f,0: SABRE PCI Bus Module

This was with my 2.6.25 .config + make oldconfig. I also built 2.6.26-rc1 with arch/sparc64/defconfig + minor tweaks to suit my machine, but that one also only found 128MB RAM.

dmesg logs and .configs are in <http://user.it.uu.se/~mikpe/linux/sparge/>

/Mikael

#1 lore raw

From: Mikael Pettersson <[email protected]>
Date: Wed, 7 May 2008 20:36:55 +0200

Quoted 5 lines

Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it reports having only 128MB RAM, while earlier kernels reported the correct amount: 256MB.

A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

Thanks for the report, I'll try to track this one down.

I have a machine configured similarly to your's, so this ought to not be too difficult to fix... I hope. :-)

#2 lore raw
In reply to Mikael Pettersson

From: Mikael Pettersson <[email protected]>
Date: Wed, 7 May 2008 20:36:55 +0200

Quoted 5 lines

Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it reports having only 128MB RAM, while earlier kernels reported the correct amount: 256MB.

A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

Try as I might I couldn't reproduce this, although I did find another bug along the way.

But that's OK, we'll add some debugging and fetch the necessary information from your machine.

The good news is that the early bootup does see all 256MB of your memory:

Top of RAM: 0x17f46000, Total RAM: 0xff40000
Memory hole size: 128MB
Entering add_active_range(0, 0, 16384) 0 entries of 256 used Entering add_active_range(0, 32768, 49023) 1 entries of 256 used Entering add_active_range(0, 49024, 49053) 2 entries of 256 used Entering add_active_range(0, 49055, 49059) 3 entries of 256 used

That "0xff40000" value is 267649024 decimal, and the size of the page ranges registered next match up.

And yet we get:

Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]

which is strange.

Between these two events there is only a handfull of bootmem allocations, which together should not total 128MB on your machine. :-)

We have some existing debugging, which I'd like you to enable on the boot command line. Simply add "numaÞbug" and that'll get some more vebose information.

Please also add the debugging patch below.

Thanks!

lib/lmb.c +4 -2
index 83287d3..3f55973 100644
@@ -19,6 +19,8 @@
 
 struct lmb lmb;
 
+#define DEBUG
+
 void lmb_dump_all(void)
 {
 #ifdef DEBUG
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);
mm/bootmem.c +30 -2
index e8fb927..78e6216 100644
@@ -242,6 +242,10 @@ __alloc_bootmem_core(struct bootmem_data *bdata, unsigned long size,
 	unsigned long node_boot_start;
 	void *node_bootmem_map;
 
+#if 1
+	printk(KERN_ERR "__alloc_bootmem_core: size(%lu) align(%lu) goal[0x%lx] limit[0x%lx]\n",
+	       size, align, goal, limit);
+#endif
 	if (!size) {
 		printk("__alloc_bootmem_core(): zero-sized request\n");
 		BUG();
@@ -369,6 +373,17 @@ found:
 	return ret;
 }
 
+static void report_range(unsigned long *start, unsigned long pfn)
+{
+	if (*start = ~0UL)
+		return;
+
+	printk(KERN_ERR "free_all_bootmem_core: Freed pfn range [0x%lx --> 0x%lx]\n",
+	       *start, pfn);
+
+	*start = ~0UL;
+}
+
 static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 {
 	struct page *page;
@@ -377,6 +392,7 @@ static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 	unsigned long i, count, total = 0;
 	unsigned long idx;
 	unsigned long *map; 
+	unsigned long debug_start_pfn = ~0UL;
 	int gofast = 0;
 
 	BUG_ON(!bdata->node_bootmem_map);
@@ -390,12 +406,17 @@ static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 	if (bdata->node_boot_start = 0 ||
 	    ffs(bdata->node_boot_start) - PAGE_SHIFT > ffs(BITS_PER_LONG))
 		gofast = 1;
+	printk(KERN_ERR "free_all_bootmem_core: pfn[0x%lx] idx[0x%lx] gofast(%d) "
+	       "node_boot_start[0x%lx]\n",
+	       pfn, idx, gofast, bdata->node_boot_start);
 	for (i = 0; i < idx; ) {
 		unsigned long v = ~map[i / BITS_PER_LONG];
 
 		if (gofast && v = ~0UL) {
 			int order;
 
+			if (debug_start_pfn = ~0UL)
+				debug_start_pfn = pfn;
 			page = pfn_to_page(pfn);
 			count += BITS_PER_LONG;
 			order = ffs(BITS_PER_LONG) - 1;
@@ -403,20 +424,27 @@ static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 			i += BITS_PER_LONG;
 			page += BITS_PER_LONG;
 		} else if (v) {
-			unsigned long m;
+			unsigned long m, debug_index;
 
 			page = pfn_to_page(pfn);
-			for (m = 1; m && i < idx; m<<=1, page++, i++) {
+			debug_index = 0;
+			for (m = 1; m && i < idx; m<<=1, page++, i++, debug_index++) {
 				if (v & m) {
+					if (debug_start_pfn = ~0UL)
+						debug_start_pfn = pfn + debug_index;
 					count++;
 					__free_pages_bootmem(page, 0);
+				} else {
+					report_range(&debug_start_pfn, pfn + debug_index);
 				}
 			}
 		} else {
+			report_range(&debug_start_pfn, pfn);
 			i += BITS_PER_LONG;
 		}
 		pfn += BITS_PER_LONG;
 	}
+	report_range(&debug_start_pfn, pfn);
 	total += count;
 
 	/*
#3 lore raw
Quoted 38 lines · David Miller

From: Mikael Pettersson <[email protected]>
Date: Wed, 7 May 2008 20:36:55 +0200

Quoted 5 lines

Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it reports having only 128MB RAM, while earlier kernels reported the correct amount: 256MB.

A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

Try as I might I couldn't reproduce this, although I did find another bug along the way.

But that's OK, we'll add some debugging and fetch the necessary information from your machine.

The good news is that the early bootup does see all 256MB of your memory:

Top of RAM: 0x17f46000, Total RAM: 0xff40000
Memory hole size: 128MB
Entering add_active_range(0, 0, 16384) 0 entries of 256 used Entering add_active_range(0, 32768, 49023) 1 entries of 256 used Entering add_active_range(0, 49024, 49053) 2 entries of 256 used Entering add_active_range(0, 49055, 49059) 3 entries of 256 used

That "0xff40000" value is 267649024 decimal, and the size of the page ranges registered next match up.

And yet we get:

Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]

which is strange.

Between these two events there is only a handfull of bootmem allocations, which together should not total 128MB on your machine. :-)

Quoted 5 lines

We have some existing debugging, which I'd like you to enable on the boot command line. Simply add "numaÞbug" and that'll get some more vebose information.

Please also add the debugging patch below.

Right, 2.6.26-rc2 plus your debugging patch and booted with numaÞbug prints the following:

[    0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
[    0.000000] PROMLIB: Root node compatible: 
[    0.000000] Linux version 2.6.26-rc2-test (mikpe@sparge) (gcc version 4.2.3) #1 Mon May 12 19:22:10 CEST 2008
[    0.000000] console [earlyprom0] enabled
[    0.000000] ARCH: SUN4U
[    0.000000] Ethernet address: 08:00:20:fd:ec:1f
[    0.000000] Found ramdisk at physical address 0x10800000, size 3683778
[    0.000000] Kernel: Using 1 locked TLB entries for main kernel image.
[    0.000000] Remapping the kernel... done.
[    0.000000] OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
[    0.000000] PROM: Built device tree with 46850 bytes of memory.
[    0.000000] bootmem_init_numa()
[    0.000000] bootmem_init_nonnuma()
[    0.000000] Top of RAM: 0x17f46000, Total RAM: 0xff40000
[    0.000000] Memory hole size: 128MB
[    0.000000] Initializing tables for non-numa.
[    0.000000] Entering add_active_range(0, 0, 16384) 0 entries of 256 used
[    0.000000] Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
[    0.000000] Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
[    0.000000] Entering add_active_range(0, 49055, 49059) 3 entries of 256 used
[    0.000000] bootmem_init_one_node(0)
[    0.000000]   init_bootmem_node(0, 3ffe, 0, bfa3)
[    0.000000]   free_bootmem_with_active_regions(0, bfa3)
[    0.000000]   trim_reserved_in_node(0)
[    0.000000]     reserve_range_in_node(nid[0],start[7ffc000],end[7ffe000]
[    0.000000]       MATCH reserving range [7ffc000:7ffe000]
[    0.000000]     reserve_range_in_node(nid[0],start[7fff040],end[8000000]
[    0.000000]       MATCH reserving range [7fff040:8001040]
[    0.000000]     reserve_range_in_node(nid[0],start[10000000],end[10359f20]
[    0.000000]       MATCH reserving range [10000000:1035a000]
[    0.000000]     reserve_range_in_node(nid[0],start[107e8cc0],end[213835c2]
[    0.000000]       MATCH reserving range [107e8cc0:21384cc0]
[    0.000000]   sparse_memory_present_with_active_regions(0)
[    0.000000] __alloc_bootmem_core: size(8192) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(32768) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(48) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(4194304) align(4194304) goal[0x7ffffffffff] limit[0x0]
[    0.000000] [0000000200000000-fffff80000400000] page_structs1072 node=0 entry=0/0
[    0.000000] __alloc_bootmem_core: size(4194304) align(4194304) goal[0x7ffffffffff] limit[0x0]
[    0.000000] [0000000200000000-fffff80000800000] page_structs1072 node=0 entry=1/0
[    0.000000] Zone PFN ranges:
[    0.000000]   Normal          0 ->    49059
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0:        0 ->    16384
[    0.000000]     0:    32768 ->    49023
[    0.000000]     0:    49024 ->    49053
[    0.000000]     0:    49055 ->    49059
[    0.000000] On node 0 totalpages: 32672
[    0.000000]   Normal zone: 335 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32337 pages, LIFO batch:7
[    0.000000] __alloc_bootmem_core: size(4096) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000]   Movable zone: 0 pages used for memmap
[    0.000000] Booting Linux...
[    0.000000] __alloc_bootmem_core: size(29) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(29) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(64) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 32337
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: ro root=/dev/sda5 numaÞbug
[    0.000000] __alloc_bootmem_core: size(32768) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] PID hash table entries: 1024 (order: 10, 8192 bytes)
[    0.000000] __alloc_bootmem_core: size(8192) align(64) goal[0x7ffffffffff] limit[0x0]
[   25.221347] clocksource: mult[28000] shift[16]
[   25.321537] clockevent: mult[66666666] shift[32]
[   25.422936] __alloc_bootmem_core: size(424) align(64) goal[0x7ffffffffff] limit[0x0]
[   25.422963] __alloc_bootmem_core: size(4000) align(64) goal[0x7ffffffffff] limit[0x0]
[   25.423055] Console: colour dummy device 80x25
[   25.910678] console handover: boot [earlyprom0] -> real [tty0]
[   26.019608] __alloc_bootmem_core: size(262144) align(64) goal[0x7ffffffffff] limit[0x0]
[   26.021182] Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
[   26.021716] __alloc_bootmem_core: size(131072) align(64) goal[0x7ffffffffff] limit[0x0]
[   26.022488] Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
[   26.022748] __alloc_bootmem_core: size(16) align(64) goal[0x7ffffffffff] limit[0x0]
[   26.026505] free_all_bootmem_core: pfn[0x0] idx[0xbfa3] gofast(1) node_boot_start[0x0]
[   26.026758] free_all_bootmem_core: Freed pfn range [0x3a --> 0x200]
[   26.031939] free_all_bootmem_core: Freed pfn range [0x600 --> 0x3ffe]
[   26.032216] free_all_bootmem_core: Freed pfn range [0x81ad --> 0x83f4]
[   26.032341] Memory: 127064k available (2032k kernel code, 800k data, 152k init) [fffff80000000000,0000000017f46000]
[   26.032566] SLUB: Genslabs, HWalign2, Order=0-3, MinObjects=0, CPUs=1, Nodes
[   26.179691] Calibrating delay using timer specific routine.. 800.75 BogoMIPS (lpj@03753)
[   26.180038] Mount-cache hash table entries: 512
[   26.182670] net_namespace: 344 bytes
[   26.183637] NET: Registered protocol family 16
[   26.197822] PCI: Probing for controllers.
[   26.201140] /pci@1f,0: SABRE PCI Bus Module
[   26.201192] /pci@1f,0: PCI IO[1fe02000000] MEM[1ff00000000]
[   26.201270] PCI: Scanning PBM /pci@1f,0
[   26.205349] ebus0: [auxio] [power] [SUNW,pll] [se] [su] [su] [ecpp] [fdthree] [eeprom] [flashprom] [SUNW,CS4231]
[   26.208785] power: Control reg at 1fff1724000
[   26.214386] SCSI subsystem initialized
[   26.214637] libata version 3.00 loaded.
[   26.230239] AUXIO: Found device at /pci@1f,0/pci@1,1/ebus@1/auxio@14,726000
[   26.230582] /pci@1f,0/pci@1,1/ebus@1/eeprom@14,0: Clock regs at 000001fff1000000
[   26.233300] NET: Registered protocol family 2
[   26.550041] IP route cache hash table entries: 2048 (order: 1, 16384 bytes)
[   26.551261] TCP established hash table entries: 8192 (order: 4, 131072 bytes)
[   26.551948] TCP bind hash table entries: 8192 (order: 3, 65536 bytes)
[   26.552300] TCP: Hash tables configured (established 8192 bind 8192)
[   26.552354] TCP reno registered
[   26.580007] NET: Registered protocol family 1
[   26.580627] Unpacking initramfs... done
[   27.455017] Freeing initrd memory: 3597k freed
[   27.455794] Mini RTC Driver
[   27.480248] msgmni has been set to 255 for ipc namespace 00000000006b07d0
[   27.481545] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[   27.481631] io scheduler noop registered
[   27.481677] io scheduler anticipatory registered (default)
[   27.481725] io scheduler deadline registered
[   27.482589] io scheduler cfq registered
[   27.483143] atyfb: 3D RAGE PRO (Mach64 GP, PQFP, PCI) [0x4750 rev 0x7c]
[   27.484393] atyfb: 4M SGRAM (1:1), 14.31818 MHz XTAL, 230 MHz PLL, 100 Mhz MCLK, 100 MHz XCLK
[   27.484829] fbcon: ATY Mach64 (fb0) is primary device
[   27.562718] Console: switching to colour frame buffer device 160x64
[   27.593689] atyfb: fb0: ATY Mach64 frame buffer device on PCI
[   27.610070] /pci@1f,0/pci@1,1/ebus@1/su@14,3083f8: Keyboard port at 1fff13083f8, irq 6
[   27.610530] /pci@1f,0/pci@1,1/ebus@1/su@14,3062f8: Mouse port at 1fff13062f8, irq 7
[   27.611497] f006159c: ttyS0 at MMIO 0x1fff1400000 (irq = 5) is a SAB82532 V3.2
[   27.612311] f006159c: ttyS1 at MMIO 0x1fff1400040 (irq = 5) is a SAB82532 V3.2
[   27.613450] Driver 'sd' needs updating - please use bus_type methods
[   27.614385] scsi0 : pata_cmd64x
[   27.615098] scsi1 : pata_cmd64x
[   27.615556] ata1: PATA max MWDMA2 cmd 0x1fe02c00000 ctl 0x1fe02c00008 bmdma 0x1fe02c00020 irq 14
[   27.615910] ata2: PATA max MWDMA2 cmd 0x1fe02c00010 ctl 0x1fe02c00018 bmdma 0x1fe02c00028 irq 14
[   27.616320] pata_cmd64x: active 10 recovery 10 setup 3.
[   27.616343] pata_cmd64x: active 10 recovery 10 setup 3.
[   27.790122] ata1.00: ATA-4: ST320420A, 3.21, max UDMA/66
[   27.790346] ata1.00: 39851760 sectors, multi 0: LBA 
[   27.790603] pata_cmd64x: active 3 recovery 1 setup 1.
[   27.790631] pata_cmd64x: active 3 recovery 1 setup 1.
[   27.830069] ata1.00: configured for MWDMA2
[   27.830306] pata_cmd64x: active 10 recovery 10 setup 3.
[   27.830329] pata_cmd64x: active 10 recovery 10 setup 3.
[   28.010111] ata2.00: ATAPI: CRD-8483B, 1.00, max UDMA/33
[   28.010367] pata_cmd64x: active 3 recovery 1 setup 1.
[   28.010392] pata_cmd64x: active 3 recovery 1 setup 1.
[   28.030077] ata2.00: configured for MWDMA2
[   28.030703] scsi 0:0:0:0: Direct-Access     ATA      ST320420A        3.21 PQ: 0 ANSI: 5
[   28.031739] sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
[   28.032111] sd 0:0:0:0: [sda] Write Protect is off
[   28.032317] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   28.032479] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   28.033106] sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
[   28.033465] sd 0:0:0:0: [sda] Write Protect is off
[   28.033676] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   28.033833] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   28.034192]  sda: sda1 sda2 sda3 sda4 sda5
[   28.040646] sd 0:0:0:0: [sda] Attached SCSI disk
[   28.041747] scsi 1:0:0:0: CD-ROM            LG       CD-ROM CRD-8483B 1.00 PQ: 0 ANSI: 5
[   28.044017] mice: PS/2 mouse device common for all mice
[   28.694315] input: Sun Type 5 keyboard as /class/input/input0
[   28.721130] input: Sun Mouse as /class/input/input1
[   28.752541] TCP cubic registered
[   28.752695] NET: Registered protocol family 17
[   29.749831] kjournald starting.  Commit interval 5 seconds
[   29.757309] EXT3-fs: mounted filesystem with ordered data mode.
[   36.303502] PCI: Enabling device: (0000:01:01.1), cmd 2
[   36.303535] sunhme.c:v3.00 June 23, 2006 David S. Miller ([email protected])
[   36.305722] eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:fd:ec:1f
[   39.701765] EXT3 FS on sda5, internal journal
[   39.919869] kjournald starting.  Commit interval 5 seconds
[   39.920286] EXT3 FS on sda1, internal journal
[   39.920311] EXT3-fs: mounted filesystem with ordered data mode.
[   39.938232] kjournald starting.  Commit interval 5 seconds
[   39.938753] EXT3 FS on sda2, internal journal
[   39.938776] EXT3-fs: mounted filesystem with ordered data mode.
[   40.628505] Adding 1048808k swap on /dev/sda4.  Priority:-1 extents:1 across:1048808k
[   46.099830] eth0: Link is up using internal transceiver at 100Mb/s, Full Duplex.

One thing that bothers me is the "Memory hole size: 128MB" message. It wasn't present in 2.6.25, and the amount reported is the exact same amount of RAM I've lost. Or maybe I'm paranoid :-)

/Mikael

#4 lore raw

From: Mikael Pettersson <[email protected]>
Date: Mon, 12 May 2008 21:06:41 +0200

Quoted 9 lines
Quoted 5 lines · David Miller

We have some existing debugging, which I'd like you to enable on the boot command line. Simply add "numaÞbug" and that'll get some more vebose information.

Please also add the debugging patch below.

Right, 2.6.26-rc2 plus your debugging patch and booted with numaÞbug prints the following:

Thanks, I'll look over this!

Quoted 3 lines

One thing that bothers me is the "Memory hole size: 128MB" message. It wasn't present in 2.6.25, and the amount reported is the exact same amount of RAM I've lost. Or maybe I'm paranoid :-)

It's completely new and harmless. You have memory at 0->128MB and at 256MB->384MB. That gap from 128MB->256MB is the hole is it talking about.

It's computed very simply, it is the difference between the highest value physical address of available memory and the total amount of available memory.

Don't let it scare you, it's 2GB on one of my machines :-)

#5 lore raw
In reply to Mikael Pettersson

From: Mikael Pettersson <[email protected]>
Date: Mon, 12 May 2008 21:06:41 +0200

Quoted 5 lines
Quoted 1 line · David Miller

Please also add the debugging patch below.

Right, 2.6.26-rc2 plus your debugging patch and booted with numaÞbug prints the following:

Hmmm... my debugging patch had this:

lib/lmb.c +0 -0
index 83287d3..3f55973 100644

...
+#define DEBUG
+

which should have resulted in lmb_dump_all() printing some userful debugging messages. I validated that it did so on my machine with the patch applied, but they appear nowhere in your logs :(

paginig_init() in arch/sparc64/mm/init.c calls lmb_analyze() them lmb_dump_all().

Those messages go out with KERN_DEBUG log level, maybe messages at that level were trimmed by your log capture for some reason?

In any event I think I know the area that's cause some kind of problem. It looks like lmb_alloc() has a case where it will reserve the wrong amount of memory, or something like that.

You can remove the debugging patch I sent you, and try this one instead. Please make sure KERN_DEBUG messages make it into the log :-)

arch/sparc64/mm/init.c +2 -0
index a9828d7..a628a99 100644
@@ -1353,6 +1353,8 @@ static void __init bootmem_init_one_node(int nid)
 
 	numadbg("bootmem_init_one_node(%d)\n", nid);
 
+	lmb_dump_all();
+
 	p = NODE_DATA(nid);
 
 	if (p->node_spanned_pages) {
lib/lmb.c +4 -2
index 83287d3..d8c84f3 100644
@@ -17,6 +17,8 @@
 
 #define LMB_ALLOC_ANYWHERE	0
 
+#define DEBUG
+
 struct lmb lmb;
 
 void lmb_dump_all(void)
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);
#6 lore raw

From: David Miller <[email protected]>
Date: Mon, 12 May 2008 15:31:43 -0700 (PDT)

Quoted 15 lines

From: Mikael Pettersson <[email protected]>
Date: Mon, 12 May 2008 21:06:41 +0200

Quoted 5 lines
Quoted 1 line · David Miller

Please also add the debugging patch below.

Right, 2.6.26-rc2 plus your debugging patch and booted with numaÞbug prints the following:

Hmmm... my debugging patch had this:

lib/lmb.c +0 -0
index 83287d3..3f55973 100644

lib/lmb.c line 21

Quoted 3 lines
lib/lmb.c +2 -0
...
+#define DEBUG
+

Nevermind I see why the messages don't show up. Hold on for a second and I'll send an updated debugging patch.

Thanks.

#7 lore raw

From: David Miller <[email protected]>
Date: Mon, 12 May 2008 15:36:28 -0700 (PDT)

Quoted 2 lines

Nevermind I see why the messages don't show up. Hold on for a second and I'll send an updated debugging patch.

Try this patch instead, thanks!

arch/sparc64/mm/init.c +2 -0
index a9828d7..a628a99 100644
@@ -1353,6 +1353,8 @@ static void __init bootmem_init_one_node(int nid)
 
 	numadbg("bootmem_init_one_node(%d)\n", nid);
 
+	lmb_dump_all();
+
 	p = NODE_DATA(nid);
 
 	if (p->node_spanned_pages) {
lib/lmb.c +4 -2
index 83287d3..f294bbc 100644
@@ -10,6 +10,8 @@
  *      2 of the License, or (at your option) any later version.
  */
 
+#define DEBUG
+
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/bitops.h>
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);
#8 lore raw
Quoted 45 lines · David Miller

From: David Miller <[email protected]>
Date: Mon, 12 May 2008 15:36:28 -0700 (PDT)

Quoted 2 lines

Nevermind I see why the messages don't show up. Hold on for a second and I'll send an updated debugging patch.

Try this patch instead, thanks!

arch/sparc64/mm/init.c +2 -0
index a9828d7..a628a99 100644
@@ -1353,6 +1353,8 @@ static void __init bootmem_init_one_node(int nid)
 
 	numadbg("bootmem_init_one_node(%d)\n", nid);
 
+	lmb_dump_all();
+
 	p = NODE_DATA(nid);
 
 	if (p->node_spanned_pages) {
lib/lmb.c +3 -1
index 83287d3..f294bbc 100644
@@ -10,6 +10,8 @@
  *      2 of the License, or (at your option) any later version.
  */
 
+#define DEBUG
+
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/bitops.h>
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);

lib/lmb.c void lmb_dump_all(void) · line 46

Quoted 8 lines
lib/lmb.c +1 -1
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);

Ok, reverting previous patch and applying this done results in:

PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
PROMLIB: Root node compatible:
Linux version 2.6.26-rc2-test (mikpe@sparge) (gcc version 4.2.3) #2 Tue May 13 18:26:56 CEST 2008 console [earlyprom0] enabled
ARCH: SUN4U
Ethernet address: 08:00:20:fd:ec:1f
Found ramdisk at physical address 0x10800000, size 3683665
lmb_dump_all:

    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x2
    reserved.size	  = 0x0
    reserved.region[0x0].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x1].base       = 0x10800000
		      .size     = 0x10b83551

Kernel: Using 1 locked TLB entries for main kernel image. Remapping the kernel... done.
OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
PROM: Built device tree with 46641 bytes of memory. bootmem_init_numa()
bootmem_init_nonnuma()
Top of RAM: 0x17f46000, Total RAM: 0xff40000
Memory hole size: 128MB
Initializing tables for non-numa.
Entering add_active_range(0, 0, 16384) 0 entries of 256 used Entering add_active_range(0, 32768, 49023) 1 entries of 256 used Entering add_active_range(0, 49024, 49053) 2 entries of 256 used Entering add_active_range(0, 49055, 49059) 3 entries of 256 used bootmem_init_one_node(0)
lmb_dump_all:

    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x4
    reserved.size	  = 0x0
    reserved.region[0x0].base       = 0x7ffc000
		      .size     = 0x2000
    reserved.region[0x1].base       = 0x7fff040
		      .size     = 0xfc0
    reserved.region[0x2].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x3].base       = 0x107e8e00
		      .size     = 0x10b9a751
  init_bootmem_node(0, 3ffe, 0, bfa3)
  free_bootmem_with_active_regions(0, bfa3)
  trim_reserved_in_node(0)
    reserve_range_in_node(nid[0],start[7ffc000],end[7ffe000]
      MATCH reserving range [7ffc000:7ffe000]
    reserve_range_in_node(nid[0],start[7fff040],end[8000000]
      MATCH reserving range [7fff040:8001040]
    reserve_range_in_node(nid[0],start[10000000],end[1035bf60]
      MATCH reserving range [10000000:1035c000]
    reserve_range_in_node(nid[0],start[107e8e00],end[21383551]
      MATCH reserving range [107e8e00:21384e00]

sparse_memory_present_with_active_regions(0)
[0000000200000000-fffff80000400000] page_structs1072 node=0 entry=0/0 [0000000200000000-fffff80000800000] page_structs1072 node=0 entry=1/0 Zone PFN ranges:
Normal 0 -> 49059
Movable zone start PFN for each node
early_node_map[4] active PFN ranges

    0:        0 ->    16384
    0:    32768 ->    49023
    0:    49024 ->    49053
    0:    49055 ->    49059

On node 0 totalpages: 32672
Normal zone: 335 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 32337 pages, LIFO batch:7
Movable zone: 0 pages used for memmap
Booting Linux...
Built 1 zonelists in Node order, mobility grouping on. Total pages: 32337
Policy zone: Normal
Kernel command line: ro root=/dev/sda5 numaÞbug
PID hash table entries: 1024 (order: 10, 8192 bytes)
clocksource: mult[28000] shift[16]
clockevent: mult[66666666] shift[32]
Console: colour dummy device 80x25
console handover: boot [earlyprom0] -> real [tty0]
Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
Memory: 127056k available (2040k kernel code, 808k data, 152k init) [fffff80000000000,0000000017f46000]
SLUB: Genslabs, HWalign2, Order=0-3, MinObjects=0, CPUs=1, Nodes
Calibrating delay using timer specific routine.. 800.75 BogoMIPS (lpj@03776) Mount-cache hash table entries: 512
net_namespace: 344 bytes
NET: Registered protocol family 16
PCI: Probing for controllers.
/pci@1f,0: SABRE PCI Bus Module
/pci@1f,0: PCI IO[1fe02000000] MEM[1ff00000000]
PCI: Scanning PBM /pci@1f,0
ebus0: [auxio] [power] [SUNW,pll] [se] [su] [su] [ecpp] [fdthree] [eeprom] [flashprom] [SUNW,CS4231]
power: Control reg at 1fff1724000
SCSI subsystem initialized
libata version 3.00 loaded.
AUXIO: Found device at /pci@1f,0/pci@1,1/ebus@1/auxio@14,726000 /pci@1f,0/pci@1,1/ebus@1/eeprom@14,0: Clock regs at 000001fff1000000
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 16384 bytes)
TCP established hash table entries: 8192 (order: 4, 131072 bytes)
TCP bind hash table entries: 8192 (order: 3, 65536 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
Unpacking initramfs... done
Freeing initrd memory: 3597k freed
Mini RTC Driver
msgmni has been set to 255 for ipc namespace 00000000006b27c8
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
atyfb: 3D RAGE PRO (Mach64 GP, PQFP, PCI) [0x4750 rev 0x7c]
atyfb: 4M SGRAM (1:1), 14.31818 MHz XTAL, 230 MHz PLL, 100 Mhz MCLK, 100 MHz XCLK
fbcon: ATY Mach64 (fb0) is primary device
Console: switching to colour frame buffer device 144x56
atyfb: fb0: ATY Mach64 frame buffer device on PCI /pci@1f,0/pci@1,1/ebus@1/su@14,3083f8: Keyboard port at 1fff13083f8, irq 6 /pci@1f,0/pci@1,1/ebus@1/su@14,3062f8: Mouse port at 1fff13062f8, irq 7
f006159c: ttyS0 at MMIO 0x1fff1400000 (irq = 5) is a SAB82532 V3.2
f006159c: ttyS1 at MMIO 0x1fff1400040 (irq = 5) is a SAB82532 V3.2
Driver 'sd' needs updating - please use bus_type methods
scsi0 : pata_cmd64x
scsi1 : pata_cmd64x
ata1: PATA max MWDMA2 cmd 0x1fe02c00000 ctl 0x1fe02c00008 bmdma 0x1fe02c00020 irq 14
ata2: PATA max MWDMA2 cmd 0x1fe02c00010 ctl 0x1fe02c00018 bmdma 0x1fe02c00028 irq 14
pata_cmd64x: active 10 recovery 10 setup 3.
pata_cmd64x: active 10 recovery 10 setup 3.
ata1.00: ATA-4: ST320420A, 3.21, max UDMA/66
ata1.00: 39851760 sectors, multi 0: LBA
pata_cmd64x: active 3 recovery 1 setup 1.
pata_cmd64x: active 3 recovery 1 setup 1.
ata1.00: configured for MWDMA2
pata_cmd64x: active 10 recovery 10 setup 3.
pata_cmd64x: active 10 recovery 10 setup 3.
ata2.00: ATAPI: CRD-8483B, 1.00, max UDMA/33
pata_cmd64x: active 3 recovery 1 setup 1.
pata_cmd64x: active 3 recovery 1 setup 1.
ata2.00: configured for MWDMA2
scsi 0:0:0:0: Direct-Access ATA ST320420A 3.21 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2 sda3 sda4 sda5
sd 0:0:0:0: [sda] Attached SCSI disk
scsi 1:0:0:0: CD-ROM LG CD-ROM CRD-8483B 1.00 PQ: 0 ANSI: 5
mice: PS/2 mouse device common for all mice
input: Sun Type 5 keyboard as /class/input/input0
input: Sun Mouse as /class/input/input1
TCP cubic registered
NET: Registered protocol family 17
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
PCI: Enabling device: (0000:01:01.1), cmd 2
sunhme.c:v3.00 June 23, 2006 David S. Miller ([email protected])
eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:fd:ec:1f
EXT3 FS on sda5, internal journal
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 1048808k swap on /dev/sda4. Priority:-1 extents:1 across:1048808k
eth0: Link is up using internal transceiver at 100Mb/s, Full Duplex.

I also tried your two lmb patches from today (had to revert this patch first), but they made no significant difference.

/Mikael

#9 lore raw

From: Mikael Pettersson <[email protected]>
Date: Tue, 13 May 2008 21:31:19 +0200

Quoted 4 lines
Quoted 1 line · David Miller

Try this patch instead, thanks!

Ok, reverting previous patch and applying this done results in:

...

Quoted 13 lines

lmb_dump_all:

    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x4
    reserved.size	  = 0x0
Quoted 8 lines
    reserved.region[0x0].base       = 0x7ffc000
		      .size     = 0x2000
    reserved.region[0x1].base       = 0x7fff040
		      .size     = 0xfc0
    reserved.region[0x2].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x3].base       = 0x107e8e00
		      .size     = 0x10b9a751

Yeah, those last two reserved regions are where your 128MB went to, as I suspected.

Thanks I'll try to figure out where to go from here.

#10 lore raw
In reply to Mikael Pettersson

From: Mikael Pettersson <[email protected]>
Date: Tue, 13 May 2008 21:31:19 +0200

Ok, Mikael, I think I figured out this bug:

Quoted 18 lines

Found ramdisk at physical address 0x10800000, size 3683665 lmb_dump_all:

    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x2
    reserved.size	  = 0x0
    reserved.region[0x0].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x1].base       = 0x10800000
		      .size     = 0x10b83551

From the very beginning your higher RAM is gone. It's correct that some memory should be reserved, for the ramdisk, but not 128MB :-)

The ramdisk is just under 4MB in size, so something is fishy here.

And indeed, I'm reserving the wrong length. Please try this patch:

sparc64: Fix lmb_reserve() args in find_ramdisk().

This fixes the missing ram regression reported by Mikael Pettersson <[email protected]>, much thanks for all of this help in diagnosing this.

The second argument to lmb_reserve() is a size, not an end address bounds.

---
 arch/sparc64/mm/init.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
arch/sparc64/mm/init.c +1 -1
index a9828d7..3c7b947 100644
@@ -768,7 +768,7 @@ static void __init find_ramdisk(unsigned long phys_base)
 		initrd_start = ramdisk_image;
 		initrd_end = ramdisk_image + sparc_ramdisk_size;
 
-		lmb_reserve(initrd_start, initrd_end);
+		lmb_reserve(initrd_start, sparc_ramdisk_size);
 
 		initrd_start += PAGE_OFFSET;
 		initrd_end += PAGE_OFFSET;
Signature · 1 line
1.5.5.1.57.g5909c
author Tested-by
#11 lore raw
Quoted 63 lines · David Miller

From: Mikael Pettersson <[email protected]>
Date: Tue, 13 May 2008 21:31:19 +0200

Ok, Mikael, I think I figured out this bug:

Quoted 19 lines

Found ramdisk at physical address 0x10800000, size 3683665 lmb_dump_all:

    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x2
    reserved.size	  = 0x0
    reserved.region[0x0].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x1].base       = 0x10800000
		      .size     = 0x10b83551
Quoted 1 line

From the very beginning your higher RAM is gone.

It's correct that some memory should be reserved, for the ramdisk, but not 128MB :-)

The ramdisk is just under 4MB in size, so something is fishy here.

And indeed, I'm reserving the wrong length. Please try this patch:

sparc64: Fix lmb_reserve() args in find_ramdisk().

This fixes the missing ram regression reported by Mikael Pettersson <[email protected]>, much thanks for all of this help in diagnosing this.

The second argument to lmb_reserve() is a size, not an end address bounds.

---
 arch/sparc64/mm/init.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
arch/sparc64/mm/init.c +1 -1
index a9828d7..3c7b947 100644
@@ -768,7 +768,7 @@ static void __init find_ramdisk(unsigned long phys_base)
 		initrd_start = ramdisk_image;
 		initrd_end = ramdisk_image + sparc_ramdisk_size;
 
-		lmb_reserve(initrd_start, initrd_end);
+		lmb_reserve(initrd_start, sparc_ramdisk_size);
 
 		initrd_start += PAGE_OFFSET;
 		initrd_end += PAGE_OFFSET;
Signature · 1 line
1.5.5.1.57.g5909c

Thanks Dave. As the dmesg diff below shows, this patch fixed my lost RAM issue. I actually have about 8KB more available now than I had with 2.6.25.

dmesg-2.6.26-rc2-b renamed from dmesg-2.6.26-rc2-a +10 -6
@@ -1,14 +1,14 @@
 PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
 PROMLIB: Root node compatible: 
-Linux version 2.6.26-rc2 (mikpe@sparge) (gcc version 4.2.3) #1 Wed May 14 22:27:26 CEST 2008
+Linux version 2.6.26-rc2 (mikpe@sparge) (gcc version 4.2.3) #2 Thu May 15 16:14:24 CEST 2008
 console [earlyprom0] enabled
 ARCH: SUN4U
 Ethernet address: 08:00:20:fd:ec:1f
-Found ramdisk at physical address 0x10800000, size 3780573
+Found ramdisk at physical address 0x10800000, size 3780646
 Kernel: Using 1 locked TLB entries for main kernel image.
 Remapping the kernel... done.
 OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
-PROM: Built device tree with 42146 bytes of memory.
+PROM: Built device tree with 42148 bytes of memory.
 bootmem_init_nonnuma()
 Top of RAM: 0x17f46000, Total RAM: 0xff42000
 Memory hole size: 128MB
@@ -25,8 +25,12 @@
       MATCH reserving range [7ffe000:8000000]
     reserve_range_in_node(nid[0],start[10000000],end[1030ed28]
       MATCH reserving range [10000000:1030ed28]
-    reserve_range_in_node(nid[0],start[107eb3c0],end[2139afdd]
-      MATCH reserving range [107eb3c0:2139afdd]
+    reserve_range_in_node(nid[0],start[10800000],end[10b9b026]
+      MATCH reserving range [10800000:10b9b026]
+    reserve_range_in_node(nid[0],start[17f2f3c0],end[17f3c000]
+      MATCH reserving range [17f2f3c0:17f3c000]
+    reserve_range_in_node(nid[0],start[17f3e000],end[17f46000]
+      MATCH reserving range [17f3e000:17f46000]
   sparse_memory_present_with_active_regions(0)
 [0000000200000000-fffff80000400000] page_structs1072 node=0 entry=0/0
 [0000000200000000-fffff80000800000] page_structs1072 node=0 entry=1/0
@@ -53,9 +57,9 @@
 console handover: boot [earlyprom0] -> real [tty0]
 Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
 Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
-Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]
+Memory: 245448k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]

...

#12 lore raw
Quoted 2 lines · Mikael Pettersson

Thanks Dave. As the dmesg diff below shows, this patch fixed my lost RAM issue. I actually have about 8KB more available

Doh! s/KB/MB/ of course.

Quoted 1 line

now than I had with 2.6.25.

#14 lore raw
In reply to Mikael Pettersson
Quoted 3 lines · Mikael Pettersson

Thanks Dave. As the dmesg diff below shows, this patch fixed my lost RAM issue. I actually have about 8KB more available now than I had with 2.6.25.

[...]

Quoted 1 line

+Memory: 245448k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]

FWIW I just happened to notice the same improvement, but the difference here is actually considerably larger (upgrade was .22 -> .26):

-Memory: 243016k available (2096k kernel code, 760k data, 96k init) [fffff80000000000,0000000017f40000] +Memory: 248928k available (2128k kernel code, 800k data, 120k init) [fffff80000000000,0000000017f40000]

Dave could really start charging for these hardware upgrades :)

Signature · 1 line
     2. That which causes joy or happiness.
#15 lore raw
In reply to Mikael Pettersson
Quoted 13 lines · Josip Rodin
Quoted 3 lines · Mikael Pettersson

Thanks Dave. As the dmesg diff below shows, this patch fixed my lost RAM issue. I actually have about 8KB more available now than I had with 2.6.25.

[...]

Quoted 1 line

+Memory: 245448k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]

FWIW I just happened to notice the same improvement, but the difference here is actually considerably larger (upgrade was .22 -> .26):

(quoted diff) +1 -1
-Memory: 243016k available (2096k kernel code, 760k data, 96k init) [fffff80000000000,0000000017f40000]
+Memory: 248928k available (2128k kernel code, 800k data, 120k init) [fffff80000000000,0000000017f40000]
Quoted 1 line

Dave could really start charging for these hardware upgrades :)

Oh, this is starting to be really fun to upgrade... a v240 said:

-Memory: 2058624k available (2288k kernel code, 840k data, 176k init) [fffff80000000000,000000103fedc000] +Memory: 2070608k available (2344k kernel code, 880k data, 192k init) [fffff80000000000,000000103fedc000]

Signature · 1 line
     2. That which causes joy or happiness.