Skip to contentSkip to conversation

log-buf-len dynamic

PATCH Andrea Arcangeli 9 participants 28 messages Sep 23, 2003 – Sep 26, 2003

Subsystems (9)
Documentation Arm (32-bit) MIPS s390 +5 more subsystems
lore
Share & tools
lore
#0 lore raw

Hi,

I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in mainline is worthless for any distributor, shipping another rpm package just for the bufsize would be way overkill.

Please backout the below (extracted from bkcvs) and apply this one instead:

	http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.22aa1/00_log-buf-len-1

BTW, not sure why but bkcvs on kernel.org is still stuck at 23pre4, I hope it's not a malfunction and just a normal delay. For now I've to use a mixture between bkcvs and patches.

---------------------
PatchSet 3787
Date: 2003/08/26 19:28:13
Author: willy
Branch: HEAD
Tag: (none)
Log:
[PATCH] make log buffer length selectable

Quoted 3 lines · Marcelo Tosatti

final:

- 2.4.22-rc4 was released as 2.4.22 with no changes.

Hi Marcelo,

as you requested, here is the log_buf_len patch for inclusion in 23-pre.

Cheers,
Willy

BKrev: 3f4ba6bdLGpAZvSSSz7LVia3xLNP6g

Members:

	ChangeSet:1.3787->1.3788 
	Documentation/Configure.help:1.220->1.221 
	arch/i386/config.in:1.47->1.48 
	kernel/printk.c:1.12->1.13 

Index: linux-2.4/Documentation/Configure.help

Documentation/Configure.help renamed from Documentation/Configure.help:1.220 +13 -0
diff -u linux-2.4/Documentation/Configure.help:1.220 linux-2.4/Documentation/Configure.help:1.221
@@ -25102,6 +25102,19 @@
   output to the second serial port on these devices.  Saying N will
   cause the debug messages to appear on the first serial port.
 
+Kernel log buffer length shift
+CONFIG_LOG_BUF_SHIFT
+  The kernel log buffer has a fixed size of :
+      64 kB (2^16) on MULTIQUAD and IA64,
+     128 kB (2^17) on S390
+      32 kB (2^15) on SMP systems
+      16 kB (2^14) on UP systems
+
+  You have the ability to change this size with this paramter which
+  fixes the bit shift of to get the buffer length (which must be a
+  power of 2). Eg: a value of 16 sets the buffer to 64 kB (2^16).
+  The default value of 0 uses standard values above.
+
 Disable pgtable cache
 CONFIG_NO_PGT_CACHE
   Normally the kernel maintains a `quicklist' of preallocated

Index: linux-2.4/arch/i386/config.in

arch/i386/config.in renamed from arch/i386/config.in:1.47 +2 -0
diff -u linux-2.4/arch/i386/config.in:1.47 linux-2.4/arch/i386/config.in:1.48
@@ -477,6 +477,8 @@
    bool '  Compile the kernel with frame pointers' CONFIG_FRAME_POINTER
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/kernel/printk.c

kernel/printk.c renamed from kernel/printk.c:1.12 +4 -0
diff -u linux-2.4/kernel/printk.c:1.12 linux-2.4/kernel/printk.c:1.13
@@ -29,6 +29,7 @@
 
 #include <asm/uaccess.h>
 
+#if !defined(CONFIG_LOG_BUF_SHIFT) || (CONFIG_LOG_BUF_SHIFT - 0 == 0)
 #if defined(CONFIG_MULTIQUAD) || defined(CONFIG_IA64)
 #define LOG_BUF_LEN	(65536)
 #elif defined(CONFIG_ARCH_S390)
@@ -37,6 +38,9 @@
 #define LOG_BUF_LEN	(32768)
 #else	
 #define LOG_BUF_LEN	(16384)			/* This must be a power of two */
+#endif
+#else /* CONFIG_LOG_BUF_SHIFT */
+#define LOG_BUF_LEN (1 << CONFIG_LOG_BUF_SHIFT)
 #endif
 
 #define LOG_BUF_MASK	(LOG_BUF_LEN-1)

---------------------
PatchSet 3862
Date: 2003/08/30 15:32:03
Author: willy
Branch: HEAD
Tag: (none)
Log:
[PATCH] Fix log buffer length issues

Quoted 1 line · Marcelo Tosatti

I'm waiting for your selectable log buffer lengh fixes :)

OK, Marcelo, here comes my proposed fix. Tom, does it look better for you ? BTW, Tom, I found the exact reason for the line which irritated you :
#if !defined(CONFIG_LOG_BUF_SHIFT) || (CONFIG_LOG_BUF_SHIFT - 0 == 0)

it was because because I was worried about CONFIG_LOG_BUG_SHIFT being defined to the empty string (a simple #define CONFIG_LOG_BUG_SHIFT is enough), as I remember it happened to me when working on it. But there's no reason here for this to happen, otherwise we would have far worse problems with values extracted from .config, so I fixed it.

I added the config line to all archs. I used copy paste, so either I broke them all, either none. I'll start a compilation on my alpha while I'm on the way home, to check. I you have questions, please do ask.

Cheers,
Willy

BKrev: 3f50b563j2T0yJ7uwm8xVJWuE9qDnQ

Members:

	ChangeSet:1.3862->1.3863 
	arch/alpha/config.in:1.28->1.29 
	arch/arm/config.in:1.24->1.25 
	arch/cris/config.in:1.18->1.19 
	arch/ia64/config.in:1.21->1.22 
	arch/m68k/config.in:1.21->1.22 
	arch/mips/config-shared.in:1.7->1.8 
	arch/parisc/config.in:1.10->1.11 
	arch/ppc/config.in:1.37->1.38 
	arch/ppc64/config.in:1.11->1.12 
	arch/s390/config.in:1.13->1.14 
	arch/s390x/config.in:1.13->1.14 
	arch/sh/config.in:1.14->1.15 
	arch/sh64/config.in:1.7->1.8 
	arch/sparc/config.in:1.19->1.20 
	arch/sparc64/config.in:1.30->1.31 
	arch/x86_64/config.in:1.10->1.11 
	kernel/printk.c:1.13->1.14 

Index: linux-2.4/arch/alpha/config.in

arch/alpha/config.in renamed from arch/alpha/config.in:1.28 +2 -0
diff -u linux-2.4/arch/alpha/config.in:1.28 linux-2.4/arch/alpha/config.in:1.29
@@ -454,6 +454,8 @@
    define_tristate CONFIG_MATHEMU y
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/arm/config.in

arch/arm/config.in renamed from arch/arm/config.in:1.24 +3 -0
diff -u linux-2.4/arch/arm/config.in:1.24 linux-2.4/arch/arm/config.in:1.25
@@ -730,6 +730,9 @@
 dep_bool '  Kernel low-level debugging functions' CONFIG_DEBUG_LL $CONFIG_DEBUG_KERNEL
 dep_bool '    Kernel low-level debugging messages via footbridge serial port' CONFIG_DEBUG_DC21285_PORT $CONFIG_DEBUG_LL $CONFIG_FOOTBRIDGE
 dep_bool '    Kernel low-level debugging messages via UART2' CONFIG_DEBUG_CLPS711X_UART2 $CONFIG_DEBUG_LL $CONFIG_ARCH_CLPS711X
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/cris/config.in

arch/cris/config.in renamed from arch/cris/config.in:1.18 +2 -0
diff -u linux-2.4/arch/cris/config.in:1.18 linux-2.4/arch/cris/config.in:1.19
@@ -272,6 +272,8 @@
   int '  Profile shift count' CONFIG_PROFILE_SHIFT 2
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 source crypto/Config.in
 source lib/Config.in
 endmenu

Index: linux-2.4/arch/ia64/config.in

arch/ia64/config.in renamed from arch/ia64/config.in:1.21 +2 -0
diff -u linux-2.4/arch/ia64/config.in:1.21 linux-2.4/arch/ia64/config.in:1.22
@@ -294,4 +294,6 @@
    bool '  Turn on irq debug checks (slow!)' CONFIG_IA64_DEBUG_IRQ
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu

Index: linux-2.4/arch/m68k/config.in

arch/m68k/config.in renamed from arch/m68k/config.in:1.21 +2 -0
diff -u linux-2.4/arch/m68k/config.in:1.21 linux-2.4/arch/m68k/config.in:1.22
@@ -556,6 +556,8 @@
    bool '  Verbose BUG() reporting' CONFIG_DEBUG_BUGVERBOSE
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/mips/config-shared.in

arch/mips/config-shared.in renamed from arch/mips/config-shared.in:1.7 +3 -0
diff -u linux-2.4/arch/mips/config-shared.in:1.7 linux-2.4/arch/mips/config-shared.in:1.8
@@ -1042,6 +1042,9 @@
 else
    int  'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32
 fi
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/parisc/config.in

arch/parisc/config.in renamed from arch/parisc/config.in:1.10 +3 -0
diff -u linux-2.4/arch/parisc/config.in:1.10 linux-2.4/arch/parisc/config.in:1.11
@@ -198,6 +198,9 @@
 
 #bool 'Debug kmalloc/kfree' CONFIG_DEBUG_MALLOC
 bool 'Magic SysRq key' CONFIG_MAGIC_SYSRQ
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/ppc/config.in

arch/ppc/config.in renamed from arch/ppc/config.in:1.37 +3 -0
diff -u linux-2.4/arch/ppc/config.in:1.37 linux-2.4/arch/ppc/config.in:1.38
@@ -563,4 +563,7 @@
 if [ "$CONFIG_GEN550" = "y" ]; then
   bool 'Support for early boot texts over serial port' CONFIG_SERIAL_TEXT_DEBUG
 fi
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu

Index: linux-2.4/arch/ppc64/config.in

arch/ppc64/config.in renamed from arch/ppc64/config.in:1.11 +3 -0
diff -u linux-2.4/arch/ppc64/config.in:1.11 linux-2.4/arch/ppc64/config.in:1.12
@@ -263,4 +263,7 @@
    dep_bool '  LKCD RLE compression' CONFIG_DUMP_COMPRESS_RLE $CONFIG_DUMP
    dep_bool '  LKCD GZIP compression' CONFIG_DUMP_COMPRESS_GZIP $CONFIG_DUMP
 fi
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu

Index: linux-2.4/arch/s390/config.in

arch/s390/config.in renamed from arch/s390/config.in:1.13 +3 -0
diff -u linux-2.4/arch/s390/config.in:1.13 linux-2.4/arch/s390/config.in:1.14
@@ -80,6 +80,9 @@
 #  bool 'Remote GDB kernel debugging' CONFIG_REMOTE_DEBUG
 #fi
 bool 'Magic SysRq key' CONFIG_MAGIC_SYSRQ
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/s390x/config.in

arch/s390x/config.in renamed from arch/s390x/config.in:1.13 +3 -0
diff -u linux-2.4/arch/s390x/config.in:1.13 linux-2.4/arch/s390x/config.in:1.14
@@ -84,6 +84,9 @@
 #  bool 'Remote GDB kernel debugging' CONFIG_REMOTE_DEBUG
 #fi
 bool 'Magic SysRq key' CONFIG_MAGIC_SYSRQ
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/sh/config.in

arch/sh/config.in renamed from arch/sh/config.in:1.14 +3 -0
diff -u linux-2.4/arch/sh/config.in:1.14 linux-2.4/arch/sh/config.in:1.15
@@ -465,6 +465,9 @@
           8 CONFIG_KGDB_DEFBITS_8"	8
    endmenu
 fi
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/sh64/config.in

arch/sh64/config.in renamed from arch/sh64/config.in:1.7 +2 -0
diff -u linux-2.4/arch/sh64/config.in:1.7 linux-2.4/arch/sh64/config.in:1.8
@@ -301,6 +301,8 @@
 bool "Debug: audit page tables on return from syscall/exception/interrupt" CONFIG_SH64_PAGE_TABLE_AUDIT
 dep_bool "Debug: report TLB fill/purge activity through /proc/tlb" CONFIG_SH64_PROC_TLB $CONFIG_PROC_FS
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source lib/Config.in

Index: linux-2.4/arch/sparc/config.in

arch/sparc/config.in renamed from arch/sparc/config.in:1.19 +2 -0
diff -u linux-2.4/arch/sparc/config.in:1.19 linux-2.4/arch/sparc/config.in:1.20
@@ -277,6 +277,8 @@
    bool '  Spinlock debugging' CONFIG_DEBUG_SPINLOCK
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/sparc64/config.in

arch/sparc64/config.in renamed from arch/sparc64/config.in:1.30 +2 -0
diff -u linux-2.4/arch/sparc64/config.in:1.30 linux-2.4/arch/sparc64/config.in:1.31
@@ -311,6 +311,8 @@
    define_bool CONFIG_MCOUNT y
 fi
 
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source crypto/Config.in

Index: linux-2.4/arch/x86_64/config.in

arch/x86_64/config.in renamed from arch/x86_64/config.in:1.10 +3 -0
diff -u linux-2.4/arch/x86_64/config.in:1.10 linux-2.4/arch/x86_64/config.in:1.11
@@ -245,6 +245,9 @@
    bool '  IOMMU leak tracing' CONFIG_IOMMU_LEAK
    bool '  Probalistic stack overflow check' CONFIG_DEBUG_STACKOVERFLOW
 fi
+
+int 'Kernel messages buffer length shift (0 = default)' CONFIG_LOG_BUF_SHIFT 0
+
 endmenu
 
 source lib/Config.in

Index: linux-2.4/kernel/printk.c

kernel/printk.c renamed from kernel/printk.c:1.13 +1 -1
diff -u linux-2.4/kernel/printk.c:1.13 linux-2.4/kernel/printk.c:1.14
@@ -29,7 +29,7 @@
 
 #include <asm/uaccess.h>
 
-#if !defined(CONFIG_LOG_BUF_SHIFT) || (CONFIG_LOG_BUF_SHIFT - 0 == 0)
+#if !defined(CONFIG_LOG_BUF_SHIFT) || (CONFIG_LOG_BUF_SHIFT == 0)
 #if defined(CONFIG_MULTIQUAD) || defined(CONFIG_IA64)
 #define LOG_BUF_LEN	(65536)
 #elif defined(CONFIG_ARCH_S390)

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#1 lore raw
Quoted 2 lines · Andrea Arcangeli

Date: 2003/08/26 19:28:13
Author: willy

Must be a different willy.

Quoted 3 lines

I added the config line to all archs. I used copy paste, so either I broke them all, either none. I'll start a compilation on my alpha while I'm on the way home, to check. I you have questions, please do ask.

... because I don't have an Alpha ;-)

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
#2 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 2 lines · Andrea Arcangeli

Date: 2003/08/26 19:28:13
Author: willy

Must be a different willy.

Quoted 3 lines

I added the config line to all archs. I used copy paste, so either I broke them all, either none. I'll start a compilation on my alpha while I'm on the way home, to check. I you have questions, please do ask.

... because I don't have an Alpha ;-)

;)

if Marcelo would be using open source code to exports the patchsets in his tree, we could fix it to add the email address along the name in the checkin logs metadata, to avoid this sort of mistakes.

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#3 lore raw
In reply to Andrea Arcangeli

Hi,

(it was from me)

Quoted 10 lines · Andrea Arcangeli

Hi,

I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in mainline is worthless for any distributor, shipping another rpm package just for the bufsize would be way overkill.

Please backout the below (extracted from bkcvs) and apply this one instead:

	http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.22aa1/00_log-buf-len-1

Well, Andrea, I've looked at your patch. I really like the dynamic size reconfiguration, but:
- now it becomes mandatory to add a command line option, just for this. It's annoying when you want to build install images for many systems. I've always considered that command line parameters should be limited to the strict minimum to have a system boot reliably (root=, console=, very few IDE/SCSI/ACPI tuning when absolutely needed) and that's all.
- what does the initial __log_buf[] become after log_buf_len_setup() ? can these 64 kB be freed or are they definitely lost ?

I think that perhaps we should merge the two things, but reconfigure them differently :

- be able to specify de DEFAULT buffer size at compile time.
- have it reconfigurable at run time with a sysctl (this could be something next to 'prink', or even a write to kmsg). This way, if you detect that your system is still loosing messages under load, you have a chance to catch them all.
- initialize the buffer with allocated memory from the beginning so that we can free it when changing the buffer size.

I can spend a few hours working with you on this if you're interested. But be assured that I know enough people who would complain about being forced to add a new boot option to their lilo.conf.

Cheers,
Willy

#4 lore raw
Quoted 37 lines · Willy Tarreau

Hi,

(it was from me)

Quoted 10 lines · Andrea Arcangeli

Hi,

I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in mainline is worthless for any distributor, shipping another rpm package just for the bufsize would be way overkill.

Please backout the below (extracted from bkcvs) and apply this one instead:

	http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.22aa1/00_log-buf-len-1

Well, Andrea, I've looked at your patch. I really like the dynamic size reconfiguration, but:
- now it becomes mandatory to add a command line option, just for this. It's annoying when you want to build install images for many systems. I've always considered that command line parameters should be limited to the strict minimum to have a system boot reliably (root=, console=, very few IDE/SCSI/ACPI tuning when absolutely needed) and that's all.
- what does the initial __log_buf[] become after log_buf_len_setup() ? can these 64 kB be freed or are they definitely lost ?

I think that perhaps we should merge the two things, but reconfigure them differently :

- be able to specify de DEFAULT buffer size at compile time.
- have it reconfigurable at run time with a sysctl (this could be something next to 'prink', or even a write to kmsg). This way, if you detect that your system is still loosing messages under load, you have a chance to catch them all.
- initialize the buffer with allocated memory from the beginning so that we can free it when changing the buffer size.

Quoted 3 lines

I can spend a few hours working with you on this if you're interested. But be assured that I know enough people who would complain about being forced to add a new boot option to their lilo.conf.

The point here is that the default must work for 99% of the userbase. Either that or the default is totally broken.

So the rest of 1% should be ok to add the command line, and they should be very happy that when they overflow again even with 128k because their cpu is too slow to keep up in klogd or whatever, they can change it to 256k without replacing kernels, some embedded platforms especially should like it. And only this 1% would be the one recompiling the kernel by themself anyways, and if they can recompile the kernel they can as well edit the defaults in kernel/printk.c without pain.

I don't buy much the lazyness argument in changing lilo.conf, the people who need this feature is a marginal part so they must be ok with the parmeter. And don't tell me that you don't pass root= to the kernel at boot. Do you want to fix that too? I do pass plenty of argumetns all the time, starting from profile=0 (and often acpi=off).

however I won't complain if you put the compile time configurator on top of my patch (that's easy) but personaly I think it's not needed, and having it dyanmic is an order of magnitude more important than having it static (again: if you can afford to recompile the kernel than you could edit kernel/printk.c in the first place without much slowdown).

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#5 lore raw

Hi Andrea,

Quoted 2 lines

The point here is that the default must work for 99% of the userbase. Either that or the default is totally broken.

You're only thinking about what distro vendors should do. If all kernels should suit their needs, then there would not be any config anymore, and everyone would have the same kernel with the same wagon of command line arguments.

Lots of people actually do recompile their kernel to change a few options from the vendor config, and that's the expected behaviour. And be it 1%, I consider that 1% of Linux users are *lots* of people.

Quoted 3 lines

And only this 1% would be the one recompiling the kernel by themself anyways, and if they can recompile the kernel they can as well edit the defaults in kernel/printk.c without pain.

I don't agree. That's what I did for a long time, and finally got bored of it. Then I wrote a patch and proposed it. Randy Dunlap had one included in 2.6, and Marcelo thought this one would be useful to lots of people, *exactly* because it avoids what you describe here, ie. changins constants by hand.

Quoted 5 lines

I don't buy much the lazyness argument in changing lilo.conf, the people who need this feature is a marginal part so they must be ok with the parmeter. And don't tell me that you don't pass root= to the kernel at boot. Do you want to fix that too? I do pass plenty of argumetns all the time, starting from profile=0 (and often acpi=off).

sorry, I don't agree with you on this. My lilo.conf contains "root=/dev/root" and "boot=/dev/mbr" on nearly all my systems, just because it's easier ton maintain, especially those which are patched remotely. You are speaking about a developer's system, and I too have lots of crap on my desktop machines, and I too am regularly annoyed because lilo tells me that 19 boot images is the max. But on production systems, you need to avoid tricks, and even more when other people manage the system after you install it.

Quoted 4 lines

however I won't complain if you put the compile time configurator on top of my patch (that's easy) but personaly I think it's not needed, and having it dyanmic is an order of magnitude more important than having it static

I said I agree on the dynamic advantage. BTW, you didn't reply me about what the statically allocated 64 kB became with your patch. Will this memory be wasted forever or is there a way to free it ? If I want to pre-allocate it dynamically with alloc_space() as you did, what can I use to free it ? is kfree() the right thing ?

Cheers,
Willy

#6 lore raw
Quoted 8 lines · Willy Tarreau

Hi Andrea,

Quoted 2 lines

The point here is that the default must work for 99% of the userbase. Either that or the default is totally broken.

You're only thinking about what distro vendors should do. If all kernels should suit their needs, then there would not be any config anymore, and everyone would have the same kernel with the same wagon of command line arguments.

this is actually not true. Many options are way too costly to recompile every time, or to keep enabled at runtime, it can't be the case for this one. My own self compiled kernel has a very stripped down .config, it's not like a distro kernel with everything included.

Quoted 2 lines

I said I agree on the dynamic advantage. BTW, you didn't reply me about what the statically allocated 64 kB became with your patch. Will this memory be

the 64k are wasted only when you use the option, if you need more than 64k it's unlikely you care about 64k lost. But I'm ok to fix it, it just looks a low priority to fix. Also since you will never use this option anyways since you don't like adding parameters to the kernel, then you don't care about fixing it either, you simply want the additional config option on top of this.

My whole argument is that being able to do it dynamic is an order of magnitude more important than being able to do it statically, no matter the command line argument and no matter ther 64k lost. You must agree with that. the amount of userbase is just not comparable.

Quoted 3 lines

wasted forever or is there a way to free it ? If I want to pre-allocate it dynamically with alloc_space() as you did, what can I use to free it ? is kfree() the right thing ?

we should simply allocate it from the bootmem pool, it's trivial to change it so the 64k aren't lost.

Not sure what I should do, personally I consider the additional .config option as configuration pollution, but since you care that much I can also change my patch not to reject yours, and to allow the two things to coexist, but I don't think it's really needed. I believe people should be forced to use the kernel params, and it's much more manageable to know all the kernels behave the same if no param is passed (otherwise you have to check the System.map to see how big the buffer is, to know if this was the right or wrong kernel during your remote maintainance, either that or you should check /proc/config.gz if you have it). It's just a mess if each kernel with the same revision number behaves differently, everything becomes less and less predictable. The number of System.map checks increases and increases before I can identify what kernel is that. Is there anybody else that has opinions on the matter?

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#7 lore raw

Andrea,

since you ask...
I consider myself a not very typical Linux user in that I am a programmer too and I (sometimes) write programs for Linux as well. And yes, I do compile my own kernels.

I think it is pretty senseless to have a configuation option for the default size of that buffer. Especially if I can, in one of the first rc scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'. The only hard requirement for that default size is that all output up till that rc script fits into a buffer of that size.

And yes, I do care about that static 64K buffer. I still have an old pentium that only has 16M. Every byte counts!

On the other hand there are the really large systems with many CPUs and such. Those might have problems with the small default. But, well, those really could #ifdef a different default based on some configuration options...

-- Jan Evert

Quoted 68 lines · Andrea Arcangeli
Quoted 9 lines · Willy Tarreau

Hi Andrea,

Quoted 2 lines

The point here is that the default must work for 99% of the userbase. Either that or the default is totally broken.

You're only thinking about what distro vendors should do. If all kernels should suit their needs, then there would not be any config anymore, and everyone would have the same kernel with the same wagon of command line arguments.

this is actually not true. Many options are way too costly to recompile every time, or to keep enabled at runtime, it can't be the case for this one. My own self compiled kernel has a very stripped down .config, it's not like a distro kernel with everything included.

Quoted 2 lines

I said I agree on the dynamic advantage. BTW, you didn't reply me about what the statically allocated 64 kB became with your patch. Will this memory be

the 64k are wasted only when you use the option, if you need more than 64k it's unlikely you care about 64k lost. But I'm ok to fix it, it just looks a low priority to fix. Also since you will never use this option anyways since you don't like adding parameters to the kernel, then you don't care about fixing it either, you simply want the additional config option on top of this.

My whole argument is that being able to do it dynamic is an order of magnitude more important than being able to do it statically, no matter the command line argument and no matter ther 64k lost. You must agree with that. the amount of userbase is just not comparable.

Quoted 3 lines

wasted forever or is there a way to free it ? If I want to pre-allocate it dynamically with alloc_space() as you did, what can I use to free it ? is kfree() the right thing ?

we should simply allocate it from the bootmem pool, it's trivial to change it so the 64k aren't lost.

Not sure what I should do, personally I consider the additional .config option as configuration pollution, but since you care that much I can also change my patch not to reject yours, and to allow the two things to coexist, but I don't think it's really needed. I believe people should be forced to use the kernel params, and it's much more manageable to know all the kernels behave the same if no param is passed (otherwise you have to check the System.map to see how big the buffer is, to know if this was the right or wrong kernel during your remote maintainance, either that or you should check /proc/config.gz if you have it). It's just a mess if each kernel with the same revision number behaves differently, everything becomes less and less predictable. The number of System.map checks increases and increases before I can identify what kernel is that. Is there anybody else that has opinions on the matter?

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#8 lore raw
Quoted 10 lines · Jan Evert van Grootheest

Andrea,

since you ask...
I consider myself a not very typical Linux user in that I am a programmer too and I (sometimes) write programs for Linux as well. And yes, I do compile my own kernels.

I think it is pretty senseless to have a configuation option for the default size of that buffer. Especially if I can, in one of the first rc scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.

having a sysctl can be an additional option (though it's tricky to implement due the needed callbacks), but the problem I guess is that most people needs a larger buf for the boot logs, so having only the sysctl would be too late...

Quoted 5 lines

The only hard requirement for that default size is that all output up till that rc script fits into a buffer of that size.

And yes, I do care about that static 64K buffer. I still have an old pentium that only has 16M. Every byte counts!

Note however that on the old 16M pentium it's unlikely that your boot logs will overflow the current 64k, so for your pentium I doubt there would be any need to use the feature. At least from my point of view the feature is meant mainly for the high end, like 16/32-way HT with 64G of ram etc.. where 64k hardly matters.

Quoted 4 lines

On the other hand there are the really large systems with many CPUs and such. Those might have problems with the small default. But, well, those really could #ifdef a different default based on some configuration options...

note the 64k are only wasted when you use the feature, there's nothing wasted if you don't use it.

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#9 lore raw
Quoted 2 lines · Andrea Arcangeli

note the 64k are only wasted when you use the feature, there's nothing wasted if you don't use it.

There's 48k wasted compared to the default 16k, because Jan will not have the choice of not allocating them with your current patch. And I'm not sure he'll be happy to follow your advice, to tap into the source file to get them back :-)

Willy

#10 lore raw
Quoted 6 lines · Willy Tarreau
Quoted 2 lines · Andrea Arcangeli

note the 64k are only wasted when you use the feature, there's nothing wasted if you don't use it.

There's 48k wasted compared to the default 16k, because Jan will not have the

48k wasted where? What arch, I thought you meant the original log_buf memory is not released, I will fix that, then the waste will be zero and there will be no advantage in having it at compile time anymore, only disavantages.

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#11 lore raw
In reply to Andrea Arcangeli
Quoted 9 lines · Andrea Arcangeli
Quoted 3 lines · Jan Evert van Grootheest

I think it is pretty senseless to have a configuation option for the default size of that buffer. Especially if I can, in one of the first rc scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.

having a sysctl can be an additional option (though it's tricky to implement due the needed callbacks), but the problem I guess is that most people needs a larger buf for the boot logs, so having only the sysctl would be too late...

IMO there are two issues:

1. On some systems it is possible to overflow the log buffer during boot and before init runs

2. On some systems, there is enough going on that klogd cannot read the log quick enough and so the log is missing lines.

IME (1) is the irritating one. I can (and have) edited the source, but it is irritating. Setting up the config option is ok, but for default kernels (e.g. a distro one) where you can't easily recompile, it's insufficient.

Proposal:

1. The default buffer size is huge: 128K or more, which should enable the initial mesages to be kept.

2. The default buffer size is modifiable on the command line, but does not have to be set there (mainly for those with v. limited RAM).

3. The log buffer can be resized smaller and bigger: using sysctl seems nice, but might a dmesg option be ok?

4. klogd and dmesg, at all times, can tell how big the buffer is and where the start and end are. That is, there is no need for the user to tell dmesg how big the buffer is.

5. Lets just do something and move on: it's not important enough to waste weeks talking about :-)

HTH,

Ruth

Signature · 2 lines
Ruth Ivimey-Cook
Software engineer and technical writer.
#12 lore raw
Quoted 30 lines · Ruth Ivimey-Cook
Quoted 8 lines · Andrea Arcangeli
Quoted 1 line · Jan Evert van Grootheest

I think it is pretty senseless to have a configuation option for the default size of that buffer. Especially if I can, in one of the first rc scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.

having a sysctl can be an additional option (though it's tricky to implement due the needed callbacks), but the problem I guess is that most people needs a larger buf for the boot logs, so having only the sysctl would be too late...

IMO there are two issues:

1. On some systems it is possible to overflow the log buffer during boot and before init runs

2. On some systems, there is enough going on that klogd cannot read the log quick enough and so the log is missing lines.

IME (1) is the irritating one. I can (and have) edited the source, but it is irritating. Setting up the config option is ok, but for default kernels (e.g. a distro one) where you can't easily recompile, it's insufficient.

Proposal:

1. The default buffer size is huge: 128K or more, which should enable the initial mesages to be kept.

2. The default buffer size is modifiable on the command line, but does not have to be set there (mainly for those with v. limited RAM).

3. The log buffer can be resized smaller and bigger: using sysctl seems nice, but might a dmesg option be ok?

Quoted 9 lines

4. klogd and dmesg, at all times, can tell how big the buffer is and where the start and end are. That is, there is no need for the user to tell dmesg how big the buffer is.

5. Lets just do something and move on: it's not important enough to waste weeks talking about :-)

HTH,

Ruth

Hi,

It may be a little over the top for 2.4, but I posted a patch for a dynamically resizable printk a while back:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105828314919697&w=2

I depends on relayfs, which I also posted at the same time.

Basically, the idea is that the static printk buffer is used only at init time for boot messages and is then discarded (so can be made very large), and after that point printk uses the relayfs channel, which is automatically grown and shrunk as necessary. Config options control the buffer sizes (LOG_BUF_SHIFT for initial static, MIN_LOG_BUF_SHIFT for the minimum dynamic) and the relay_open() call controls the maximum dynamic size and other channel characteristics.

Changes I'm currently making to relayfs should simplify the patch significantly and fix some other problems, but I thought I'd point it out as an alternative approach.

Tom

#13 lore raw
Quoted 6 lines · Tom Zanussi

It may be a little over the top for 2.4, but I posted a patch for a dynamically resizable printk a while back:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105828314919697&w=2

I depends on relayfs, which I also posted at the same time.

I don't think we can merge this in 2.4 but it looks good for 2.6. However note you can't close the overflow issue, you must still have an high limit, but you can make lots more reliable that way, it's similar to netlink anyways, that uses dynamic skb to avoid wasting ram. But it's too invasive for 2.4, just to make a comparison look what my patch looks like (it will get a bit more complicated because of the request not to waste the 64k, you know we throw some hundred megs in the per-cpu freelists so nobody could notice the 64k lost in the 16-way..):

kernel/printk.c +40 -5
diff -urNp --exclude CVS --exclude BitKeeper x-ref/kernel/printk.c x/kernel/printk.c
@@ -30,13 +30,13 @@
 #include <asm/uaccess.h>
 
 #if defined(CONFIG_MULTIQUAD) || defined(CONFIG_IA64)
-#define LOG_BUF_LEN	(65536)
+#define __LOG_BUF_LEN	(65536)
 #elif defined(CONFIG_ARCH_S390)
-#define LOG_BUF_LEN	(131072)
+#define __LOG_BUF_LEN	(131072)
 #elif defined(CONFIG_SMP)
-#define LOG_BUF_LEN	(32768)
+#define __LOG_BUF_LEN	(32768)
 #else	
-#define LOG_BUF_LEN	(16384)			/* This must be a power of two */
+#define __LOG_BUF_LEN	(16384)			/* This must be a power of two */
 #endif
 
 #define LOG_BUF_MASK	(LOG_BUF_LEN-1)
@@ -78,7 +78,9 @@ struct console *console_drivers;
  */
 static spinlock_t logbuf_lock = SPIN_LOCK_UNLOCKED;
 
-static char log_buf[LOG_BUF_LEN];
+static char __log_buf[__LOG_BUF_LEN];
+static char * log_buf = __log_buf;
+static int LOG_BUF_LEN = __LOG_BUF_LEN;
 #define LOG_BUF(idx) (log_buf[(idx) & LOG_BUF_MASK])
 
 /*
@@ -151,6 +153,39 @@ static int __init console_setup(char *st
 
 __setup("console=", console_setup);
 
+static int __init log_buf_len_setup(char *str)
+{
+	unsigned long size = simple_strtoul(str, &str, 0);
+
+	if (size & (size-1))
+		printk("log_buf_len: invalid size - needs a power of two\n");
+
+	spin_lock_irq(&logbuf_lock);
+	if (size > LOG_BUF_LEN) {
+		unsigned long start, end, dest_idx, offset;
+
+		LOG_BUF_LEN = size;
+		log_buf = alloc_exact(LOG_BUF_LEN);
+
+		offset = start = min(con_start, log_start);
+		end = log_end;
+		dest_idx = 0;
+		while (start != end) {
+			log_buf[dest_idx] = __log_buf[start & (__LOG_BUF_LEN - 1)];
+			start++;
+			dest_idx++;
+		}
+		log_start -= offset;
+		con_start -= offset;
+		log_end -= offset;
+	}
+	spin_unlock_irq(&logbuf_lock);
+
+	return 1;
+}
+
+__setup("log_buf_len=", log_buf_len_setup);
+
 /*
  * Commands to do_syslog:
  *

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#14 lore raw
Quoted 2 lines · Andrea Arcangeli

I don't think we can merge this in 2.4 but it looks good for 2.6.

I think so: doing it via a __setup parameter is much smarter than requiring a rebuild.

kernel/printk.c static int __init console_setup(char *st · line 161

Quoted 6 lines
kernel/printk.c +6 -0
+static int __init log_buf_len_setup(char *str)
+{
+	unsigned long size = simple_strtoul(str, &str, 0);
+
+	if (size & (size-1))
+		printk("log_buf_len: invalid size - needs a power of two\n");

You need to either panic or return here.

#15 lore raw
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Andrea Arcangeli

I don't think we can merge this in 2.4 but it looks good for 2.6.

I think so: doing it via a __setup parameter is much smarter than requiring a rebuild.

agreed. That's why I also preferred my simple hack. After I fix the 64k memory loss, I will see no disavantages in the kernel boot time parameter compared to the compile time option.

Quoted 9 lines

Review

Quoted 7 lines
(quoted diff) +6 -0
+static int __init log_buf_len_setup(char *str)
+{
+	unsigned long size = simple_strtoul(str, &str, 0);
+
+	if (size & (size-1))
+		printk("log_buf_len: invalid size - needs a power of two\n");

You need to either panic or return here.

woops sorry, that was the old version ;) it had other bugs too, the new one is this one (Marcelo please don't merge the old version, I thought it was just uptodate in 22aa1 but it was not, the new version wasn't in the ftp site yet infact)

Also I have to fix the request for the 64k memory loss reported by Willy first.

thanks for the review!

kernel/printk.c +46 -4
diff -urNp --exclude CVS --exclude BitKeeper ul-ref/kernel/printk.c ul/kernel/printk.c
@@ -27,15 +27,16 @@
 #include <linux/interrupt.h>			/* For in_interrupt() */
 #include <linux/config.h>
 #include <linux/delay.h>
+#include <linux/bootmem.h>
 
 #include <asm/uaccess.h>
 
 #if defined(CONFIG_MULTIQUAD) || defined(CONFIG_IA64)
-#define LOG_BUF_LEN	(65536)
+#define __LOG_BUF_LEN	(65536)
 #elif defined(CONFIG_ARCH_S390) || defined(CONFIG_PPC64) || defined(CONFIG_PPC)
-#define LOG_BUF_LEN	(131072)
+#define __LOG_BUF_LEN	(131072)
 #else	
-#define LOG_BUF_LEN	(32768)			/* This must be a power of two */
+#define __LOG_BUF_LEN	(32768)			/* This must be a power of two */
 #endif
 
 #define LOG_BUF_MASK	(LOG_BUF_LEN-1)
@@ -78,7 +79,9 @@ struct console *console_drivers;
  */
 static spinlock_t logbuf_lock = SPIN_LOCK_UNLOCKED;
 
-static char log_buf[LOG_BUF_LEN];
+static char __log_buf[__LOG_BUF_LEN];
+static char * log_buf = __log_buf;
+static int LOG_BUF_LEN = __LOG_BUF_LEN;
 #define LOG_BUF(idx) (log_buf[(idx) & LOG_BUF_MASK])
 
 /*
@@ -165,6 +168,45 @@ static int __init console_setup(char *st
 
 __setup("console=", console_setup);
 
+static int __init log_buf_len_setup(char *str)
+{
+	unsigned long size = memparse(str, &str);
+
+	if (size > LOG_BUF_LEN) {
+		unsigned long start, dest_idx, offset;
+		char * new_log_buf;
+
+		new_log_buf = alloc_bootmem(size);
+		if (!new_log_buf) {
+			printk("log_buf_len: allocation failed\n");
+			goto out;
+		}
+
+		spin_lock_irq(&logbuf_lock);
+		LOG_BUF_LEN = size;
+		log_buf = new_log_buf;
+
+		offset = start = min(con_start, log_start);
+		dest_idx = 0;
+		while (start != log_end) {
+			log_buf[dest_idx] = __log_buf[start & (__LOG_BUF_LEN - 1)];
+			start++;
+			dest_idx++;
+		}
+		log_start -= offset;
+		con_start -= offset;
+		log_end -= offset;
+		spin_unlock_irq(&logbuf_lock);
+
+		printk("log_buf_len: %d\n", LOG_BUF_LEN);
+	}
+ out:
+
+	return 1;
+}
+
+__setup("log_buf_len=", log_buf_len_setup);
+
 /*
  * Commands to do_syslog:
  *

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#16 lore raw
Quoted 2 lines · Andrea Arcangeli

+__setup("log_buf_len=", log_buf_len_setup);

Specifying this in units of bytes is a pain.

How about we make it "log_buf_kbytes", and multiply it by 1024?

#17 lore raw
Quoted 7 lines · Andrew Morton
Quoted 2 lines · Andrea Arcangeli

+__setup("log_buf_len=", log_buf_len_setup);

Specifying this in units of bytes is a pain.

How about we make it "log_buf_kbytes", and multiply it by 1024?

note: that's a memparse, that's not unit of bytes, you can wite 256k or 1M or 2m just now and it'll just work

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#18 lore raw
In reply to Jan Evert van Grootheest

[replying to both Jan and Andrea at once]

-- Jan:

Quoted 5 lines

I think it is pretty senseless to have a configuation option for the default size of that buffer. Especially if I can, in one of the first rc scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'. The only hard requirement for that default size is that all output up till that rc script fits into a buffer of that size.

I totally agree with this possibility, and it's one I proposed too. I only proposed to keep the default size in order to be able to catch all boot messages, but I think that if we make this buffer hot-resizable (and a sysctl is good for this), then we could systematically pre-allocate 64k at boot which should be enough for boot messages (16k is too short), and could be reduced if needed by a write to the sysctl. This way, the config option can go away.

Quoted 2 lines

And yes, I do care about that static 64K buffer. I still have an old pentium that only has 16M. Every byte counts!

My previous firewall add 8M, and I second your statement : every byte counts !

Quoted 11 lines
Quoted 10 lines · Andrea Arcangeli
Quoted 4 lines

You're only thinking about what distro vendors should do. If all kernels should
suit their needs, then there would not be any config anymore, and everyone would have the same kernel with the same wagon of command line arguments.

this is actually not true. Many options are way too costly to recompile every time, or to keep enabled at runtime, it can't be the case for this one. My own self compiled kernel has a very stripped down .config, it's not like a distro kernel with everything included.

I think you're mistaken about what I meant. I didn't mean that I want to recompile each kernel with a different option, I meant that for a large set of identical systems, it's more convenient to have a single, stripped down .config with the most default values, and propagate this same kernel everywhere, than having to duplicate special kernel parameters in every lilo.conf.

Quoted 2 lines
Quoted 2 lines

the 64k are wasted only when you use the option, if you need more than 64k it's unlikely you care about 64k lost.

Absolutely, my concern was most about the case where I need less than 64k and I have too few memory to agree to waste them.

Quoted 5 lines
Quoted 5 lines

But I'm ok to fix it, it just
looks a low priority to fix. Also since you will never use this option anyways since you don't like adding parameters to the kernel, then you don't care about fixing it either, you simply want the additional config option on top of this.

:-)
Andrea, I'm not *that* selfish ! If I did this only for me, I wouldn't even post it here and risk complaints from people who ask me to add other archs too. I can live with this as a patch in my kernel, next to your VM and others. It's just that I felt a need for it on production machines and thought there were potentially other people in my case. In the same way, I could say that you're wrong in believing that everyone wants to add a boot option, but I may be wrong WRT this. That's not the problem. If I asked how to fix it, it's because I'm interested in doing it myself, but I'm clueless about how to do it (don't know how to allocate boot mem, don't know how to free it, don't know if I can reallocate new mem further and free it again by the same means).

Quoted 4 lines
Quoted 4 lines

My whole argument is that being able to do it dynamic is an order of magnitude more important than being able to do it statically, no matter the command line argument and no matter ther 64k lost. You must agree with that. the amount of userbase is just not comparable.

I said I did agree with that, but it's not the same feature. And concerning the amount of userbase, I agree it's not comparable... I don't imagine desktop users adding "logbuflen=1048576" to their boot command line, nor can I imagine them recompiling their kernel changing this option.

Quoted 5 lines
Quoted 5 lines

Not sure what I should do, personally I consider the additional .config option as configuration pollution, but since you care that much I can also change my patch not to reject yours, and to allow the two things to coexist, but I don't think it's really needed. I believe people should be forced to use the kernel params,

^^^^^^^^^^^^^^^^^^^
why do you want to force people to do something the way you think is best ? I personnaly think that both solutions are a convenient way of achieving the same goal for different people. Please let them choose.

Quoted 2 lines
Quoted 2 lines

just a mess if each kernel with the same revision number behaves differently, everything becomes less and less predictable.

nothing new here. There are already many options which will make your kernel behave differently : CPU/ACPI/APM/SCSI/IDE DMA by default/Shutdown Watchdog on close, to name a few... That's why many of us add a version suffix to production kernels identifying a known .config.

Quoted 2 lines
Quoted 2 lines

The number of System.map checks increases and increases before I can identify what kernel is that.

Then don't rely on the default value for your kernels, but only on the boot option, and the problem is gone. Let us all have the choice. I insist that your patch is appealing, but it is not the only solution and does not cover all needs, nor does mine.

Cheers,
Willy

#19 lore raw
Quoted 1 line · Willy Tarreau

why do you want to force people to do something the way you think is best ?

because it gives you no real disavantage to pass the parameter in grub. You've to set root=/dev/root anyways, this way you would simply need to add the parameter for log buf as well. I've always to add lots of other stuff anyways, including vga= profile= etc..

Quoted 2 lines

I personnaly think that both solutions are a convenient way of achieving the same goal for different people. Please let them choose.

they can already choose with the parameter at boot, it's not that I don't let them choose.

You simply want *two* different APIs, where one is worhtless and obsoleted by the more powerful one.

That's why I think it's useless to have *two* APIs where one overlaps on the other *completely*.

My API is good for everyone, yours is not, I don't see why we've to keep both when nobody depends on yours yet.

Quoted 2 lines

your patch is appealing, but it is not the only solution and does not cover all needs, nor does mine.

I can easily fix the 64k ram waste, after that I can't see why my API can't cover you needs. The init code even gets released because it's __init.

I just don't buy the fact that you don't like to pass params to the kernel because you already do, you have to or it won't boot on a system different than the one that you compiled the kernel.

If you don't like to pass buf_log_len= then how can you be ok to pass root=/dev/root on all your thousand of lilo.conf in production? How can root=/dev/root be remotely acceptable if buf_log_len= can't be as well added to your thousand of distributed lilo.confs for your all images out there? This makes no sense at all to me.

Yeah, I'm forcing you to waste a dozen bytes of disk space in /etc/lilo.conf, you can definitely claim that, but I don't that as an argument either (delete the spaces/tabs and you may save more). And maybe thanks to this you won't run into the number of kernel images limitation anymore.

I'm deinitely against worthless APIs in the kernel, when more powerful one exists, and they give you zero disavantage (I will fix the 64k waste don't worry, I definitely agree that such bit must be fixed to make everyone happy).

As for decreasing the buf size, my option will allow you to decrease it too after I fix it! While the shift only let you increase the buf size, not decrease. So you should prefer the dynamic API too.

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#20 lore raw

[Cc: list stripped down]

Quoted 2 lines · Andrea Arcangeli

You simply want *two* different APIs, where one is worhtless and obsoleted by the more powerful one.

I never said I wanted *two* of them. I proposed first a fairly non-intrusive one which doesn't add any line of code, just defines, and managed to reunify architectures which used different values by default.

*You* proposed a second one. I have no objections against it, as I said, since I certainly understand that some people would prefer to use it. But *you* decided that mine was totally unusable and obsoleted by yours, reason for it to be removed. And this is *that* I objected to. Since the can both coexist and fit different people's needs without adding a byte of code (at least for mine), I don't see the goal in eliminating it, deciding unilaterally that nobody will use it.

I repeat, at the end, I don't mind. I have the patch and can live with one more patch, as I did before. It's just the fact that you decide for everyone that always adding a command line option is more convenient than a once-for-all fixed config option.

Quoted 1 line

My API is good for everyone, yours is not

I'm impressed that you know so many people. I know that mine at least satisfies a few collegues, customers, and I. So I deduced that it might be useful to others too. Even Marcelo thought the same a time ago.

Quoted 1 line

I don't see why we've to keep both when nobody depends on yours yet.

Nobody depends on mine, since it doesn't change defaults nor semantics. Perhaps some people depend on yours *not* being in. You changed LOG_BUF_LEN from a define to a static, and THAT could break other external patches. BTW, I've just checked 2.4.22aa1, I noticed that you removed kmsgdump, is it because it doesn't apply, compile or work anymore since your API change ?

Honnestly, I simply took a conservative approach. My proposal not being what will save the world, I'm 100% with you on that. Mine being useless to everyone, I'm only 20% with you on that. Yours being "good for everyone", I'm only 20% with you on that too, and perhaps less since it might break some compatibility. Yours *is* interesting to some aspects, but is not an equivalent.

Quoted 3 lines

I just don't buy the fact that you don't like to pass params to the kernel because you already do, you have to or it won't boot on a system different than the one that you compiled the kernel.

Believe it or not, there are people who don't like to pass params to the kernel. I just told a friend while I was typing this mail, and he replied me "he must be joking !". So at least, we're two on earth.

Quoted 5 lines

Yeah, I'm forcing you to waste a dozen bytes of disk space in /etc/lilo.conf, you can definitely claim that, but I don't that as an argument either (delete the spaces/tabs and you may save more). And maybe thanks to this you won't run into the number of kernel images limitation anymore.

OK Andrea, you're in a bad mood today. You're trying to prove me crazy. I won't fight. You're the kernel hacker. Not me. I have never been proud of putting a patch in the kernel and never will be. I'm more ashamed when I send an obvious "fix" which breaks someone's setup. Your attitude today seems to be the opposite. Your patch is useful, I won't contest that. It won't annoy anyone, I'll let others report it to you themselves, I've done my part of it. But they will waste their time since you don't want to listen and will try to sell your patch anyway because "it's good for everyone" as you stated it.

Quoted 2 lines

I'm deinitely against worthless APIs in the kernel, when more powerful one exists, and they give you zero disavantage

There are people out there who are against worthless lines in config files, and against APIs which break other patches.

Quoted 3 lines

As for decreasing the buf size, my option will allow you to decrease it too after I fix it! While the shift only let you increase the buf size, not decrease. So you should prefer the dynamic API too.

I don't see why you say that ! Set LOG_BUF_SHIFT to 10 and you'll have 1kB !

Have a good night anyway,
Willy

#21 lore raw
Quoted 5 lines · Willy Tarreau
Quoted 1 line

My API is good for everyone, yours is not

I'm impressed that you know so many people. I know that mine at least satisfies a few collegues, customers, and I. So I deduced that it might be useful to others too. Even Marcelo thought the same a time ago.

I know your patch was just very good for many people, but not for everyone. I really didn't want to say your patch didn't make any good, I acknowledge it was just very good.

Here I need an argument from you that explains me why should I work further on the current code to make the config option still usable. I see no reason to spend time in that effort because the config option will provide no benefit compared to the kernel parameter.

If you can explain me _why_ you don't want to pass the kernel parameter than you will convince me, if it's just that "you don't want it" then I can't buy that. Give me a valid argument and I will have a reason to retain the config option.

I definitely agree my patch (btw, I posted the last one that had a few bugs too) needs fixing to release the 64k of ram, and to allow a smaller bufsize too (the latter will happen automatically while addressing the former)

The fact you don't want to touch the lilo.conf doesn't sound to me. Especially with lilo (not grub) you've to run lilo anyways every time you replace the kernel, so a simple script adding the parameter in every lilo.conf sounds very easy to provide (you can add it in all kernels, the old ones will ignore it).

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#22 lore raw
Quoted 3 lines · Andrea Arcangeli

I know your patch was just very good for many people, but not for everyone. I really didn't want to say your patch didn't make any good, I acknowledge it was just very good.

It's not about it to be good or not, but be useful. There's a difference. When two things are good and do nearly the same, you get the best of them and that's all. When two things are useful and nearly identical, there's a huge potential that one is useful to some people, and the other one to other people. I'm certain I would use your patch on my desktop PC (if it doesn't break kmsgdump as it seems, though), because it will allow me to easily extend my log buffer when I get repetitive oopses. But most probably not on remote systems.

Quoted 4 lines

I definitely agree my patch (btw, I posted the last one that had a few bugs too) needs fixing to release the 64k of ram, and to allow a smaller bufsize too (the latter will happen automatically while addressing the former)

That doesn't matter, I'm fairly certain you will fix these issues, they are details since the patch is at its early stage.

Quoted 5 lines

The fact you don't want to touch the lilo.conf doesn't sound to me. Especially with lilo (not grub) you've to run lilo anyways every time you replace the kernel, so a simple script adding the parameter in every lilo.conf sounds very easy to provide (you can add it in all kernels, the old ones will ignore it).

Yes it's the real reason, and I think that others will second me on this one. You're seeing the benefit of *changing* the value on a kernel compile once. I see the benefit of *deploying* a same kernel to a lot of remote hosts while touching the fewest possible files. Sending a pre-configured kernel and running lilo is already a lot.

I've had private discussions off-list with a few other people who don't agree on the *obligation* to enter a new option on the command line because they will not be able to dynamically generate their boot files anymore and will have to add a config option to them just for this. Believe me, when you remotely and blindly upgrade systems with commands such as :

   for host in to_update/*; do update $host && mv $host up_to_date/ ; done

*not* having to touch a boot file is really recomforting. It's not funny to remotely run 'ed' scripts on lilo.conf when you know that there's nobody to replace the box if you fail because one of the file had once been hand-edited and is slightly different. May be you'll say "well, simply prepare all your lilo.conf, check them all then send them". If you think this, it's because you obviously never felt your heart accelerate and beat very loud during these operations and cannot understand why some people prefer to keep these delicate upgrades as trivial as possible. When I said to you that I use root=/dev/root and boot=/dev/mbr on all these hosts, it's exactly not to have to touch these sensible files.

One of us told me that he will have to rebuild his BASE package just to integrate this change, and deploy it to 1200 hosts around the world. When I see how I worry for a few tens, I think he'd better revert your patch and stay on the previous one. Think that some of these people are still running 2.2 because they don't trust 2.4 enough to risk a remote upgrade.

Andrea, I won't blame you for not understanding that some people are afraid of doing certain operations remotely. Perhaps you'll think that I'm totally stupid. That's not a problem for me. I just wanted to let you know that this sort of changes will be a regression for a few of us (mainly the few who've been using the previous patch).

As I told you the first time, if I had the choice for a dynamic buffer resizement, I'd be all for a sysctl so that I wouldn't need to touch any boot file, and that would allow to resize a live system without needing a reboot. Then, simply pre-initialize a 64kB buffer so that no boot message gets lost, and the init scripts adjust the size to what is needed.

I repeat my proposal. I agree to spend a few of my time to work with you in this direction, because I know it is a useful feature. But please don't make it more difficult for people who already have a hard time.

Thanks,
Willy

#23 lore raw
Quoted 2 lines · Willy Tarreau

because you obviously never felt your heart accelerate and beat very loud during these operations and cannot understand why some people prefer to keep

well, I kind of remeber that ;)

So ok, that sounds a good enough argument to keep the old patch too, despite you may be very few people using this.

But let's do it only in 2.4, 2.6 should have it only dynamic.

clearly if you can try to go 2.6 (possibly using a wrong .config) then you can risk editing the lilo/grub file as well. And with 2.6, with the printk rewrite that shrinks and grows the buffer, you should have no need of the parameter any more.

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk
#24 lore raw
Quoted 8 lines · Andrea Arcangeli
Quoted 2 lines · Willy Tarreau

because you obviously never felt your heart accelerate and beat very loud during these operations and cannot understand why some people prefer to keep

well, I kind of remeber that ;)

So ok, that sounds a good enough argument to keep the old patch too, despite you may be very few people using this.

Thanks for your understanding ;-)

Quoted 6 lines

But let's do it only in 2.4, 2.6 should have it only dynamic.

clearly if you can try to go 2.6 (possibly using a wrong .config) then you can risk editing the lilo/grub file as well. And with 2.6, with the printk rewrite that shrinks and grows the buffer, you should have no need of the parameter any more.

I have not tried it remotely yet, but I may adapt to a new behaviour or write a patch :-)

Cheers,
Willy

#25 lore raw
In reply to Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli

+#if !defined(CONFIG_LOG_BUF_SHIFT) || (CONFIG_LOG_BUF_SHIFT - 0 == 0)

It shouldn't have an empty string value anyway, so this is just:
#if !CONFIG_LOG_BUF_SHIFT

That handles both !defined and !0.

Signature · 2 lines
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer
#26 lore raw
In reply to Andrea Arcangeli
Quoted 5 lines · Andrea Arcangeli

Hi,

I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in mainline is worthless for any distributor, shipping another rpm package just for the bufsize would be way overkill.

Andrea,

As Willy stated previously this is useful for people who want to change the log buf size without having to change the code manually, and I think this is a useful and non intrusive change.

Do you see any problem with this?

#27 lore raw
Quoted 8 lines · On Thu, Sep 25, 2003 at 02:40:15PM +0100…
Quoted 5 lines · Andrea Arcangeli

Hi,

I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in mainline is worthless for any distributor, shipping another rpm package just for the bufsize would be way overkill.

Quoted 7 lines

Andrea,

As Willy stated previously this is useful for people who want to change the log buf size without having to change the code manually, and I think this is a useful and non intrusive change.

Do you see any problem with this?

as wrote in a later email, technically the dynamic solution obsoletes the static one, but only as far as the kernel is concerned.

It happens that to enable the dynamic solution you've to change 1 line to lilo.conf/grub.lst. since 2.4 is already being used in production distributed scenarios, there is at least a psycology value in providing a static solution that doesn't involve lilo.conf changes, so people can deploy the feature without having to edit conf files at all (with grub they don't need to run lilo either).

So I will rewrite my patch in a way that doesn't reject the static setting for next -aa (together with some other tons of pending things, but the vm merge is already completed on my tree, so I'm in sync again with pre5 on that side and that was top priority to fixup to be sure nothing was missing, thanks for merging the watermarks already btw ;).

Andrea - If you prefer relying on open source software, check these links:

	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/
	    svn://svn.kernel.org/linux-2.[46]/trunk