Linux 2.4.32-rc2

Hi,

Here goes the second release candidate for v2.4.32.

Quoted 4 lines · Marcelo Tosatti

Hi,

Here goes the second release candidate for v2.4.32.

Not break half a dozen boxen here :o) Just a compile, boot, net login.
http://bugsplatter.mine.nu/test/linux-2.4/

Grant.

Hi Marcelo, Hi Grant,

Quoted 5 lines · Grant Coady

Quoted 4 lines · Marcelo Tosatti

Hi,

Here goes the second release candidate for v2.4.32.

Quoted 4 lines

Not break half a dozen boxen here :o) Just a compile, boot, net login.
http://bugsplatter.mine.nu/test/linux-2.4/

Grant.

Using it to assemble and build 2.4.31-hf7 on athlon-SMP, and everything is fine so far. Now building it on sparc64-SMP and alpha. Marcelo, will you release this one as 2.4.32, or do you still have patches to be added ?

Hi Marcelo, Hi Grant,

Quoted 5 lines · Grant Coady

Quoted 4 lines · Marcelo Tosatti

Hi,

Here goes the second release candidate for v2.4.32.

Quoted 4 lines

Not break half a dozen boxen here :o) Just a compile, boot, net login.
http://bugsplatter.mine.nu/test/linux-2.4/

Grant.

Using it to assemble and build 2.4.31-hf7 on athlon-SMP, and everything is fine so far. Now building it on sparc64-SMP and alpha. Marcelo, will you release this one as 2.4.32, or do you still have patches to be added ?

There is nothing else pending for v2.4.32 on my part.

Will wait a couple of days and tag it as v2.4.32.

Marcelo, Willy,

Quoted 3 lines

There is nothing else pending for v2.4.32 on my part.

Will wait a couple of days and tag it as v2.4.32.

I'm checking on some IPVS related code "inconsistency" regarding the latest patches sent by Julian and a netfilter reference counting issue on SMP. Could you hold off until Sunday with the release, please. If I don't report back by then with either a bug report or bugfix, release it ;).

Willy, if you have time, could you check your non-i386 boxes with a 2.95.x compiled 2.4.x kernel, with IPVS enabled?

Also I checked back
with your hf7 series patches and I should like to note that the IPVS related patches have been merged upstream with Marcelo for 2.4.32-rc2.

Thanks guys,
Roberto Nibali, ratz

Signature · 2 lines

echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

Quoted 2 lines

Willy, if you have time, could you check your non-i386 boxes with a 2.95.x compiled 2.4.x kernel, with IPVS enabled?

Yes, no problem, but you'll have to tell me what to test ! (a config or script will save me some time). I have a Sun Ultra60 (ultrasparc SMP) which matches your description. I just have a doubt about gcc-2.95 availability on this box, I know I have a 3.3.6, do you think that the problem is gcc-related (too strong optimization or de-inlining, etc) ?

Please keep us informed when you have more info.

Bonjour Willy,

Quoted 8 lines

Quoted 2 lines

Willy, if you have time, could you check your non-i386 boxes with a 2.95.x compiled 2.4.x kernel, with IPVS enabled?

Yes, no problem, but you'll have to tell me what to test ! (a config or script will save me some time). I have a Sun Ultra60 (ultrasparc SMP) which matches your description. I just have a doubt about gcc-2.95 availability on this box, I know I have a 3.3.6, do you think that the problem is gcc-related (too strong optimization or de-inlining, etc) ?

At least following should be set, the rest you can leave to your gusto:

CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BUS=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_MMCONFIG=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SYSTEM=y

CONFIG_IP_VS=m
CONFIG_IP_VS_DEBUG=y
CONFIG_IP_VS_TAB_BITS=12
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_HPRIO=m
CONFIG_IP_VS_FTP=m

One issue is a possible C99'ism in the last IPVS patch. If you find time, please have a 2.95.x compiler installed.

Another thing that could fail is if you additionally set

CONFIG_ACPI_FAN=m

and compile with CFLAGS="-g -ggdb"

Quoted 1 line

Please keep us informed when you have more info.

I will, and I will get more details, as time permits. My beef with the IPVS code seems to be wrong, the code works as expected so far. I'm stress-testing it though until Sunday on a 4GB Dual P4 Xeon with HT combo.

Bonjour Willy,

Quoted 8 lines

Quoted 2 lines

Willy, if you have time, could you check your non-i386 boxes with a 2.95.x compiled 2.4.x kernel, with IPVS enabled?

Yes, no problem, but you'll have to tell me what to test ! (a config or script will save me some time). I have a Sun Ultra60 (ultrasparc SMP) which matches your description. I just have a doubt about gcc-2.95 availability on this box, I know I have a 3.3.6, do you think that the problem is gcc-related (too strong optimization or de-inlining, etc) ?

At least following should be set, the rest you can leave to your gusto:

CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BUS=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_MMCONFIG=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SYSTEM=y

CONFIG_IP_VS=m
CONFIG_IP_VS_DEBUG=y
CONFIG_IP_VS_TAB_BITS=12
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m

CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_HPRIO=m
CONFIG_IP_VS_FTP=m

One issue is a possible C99'ism in the last IPVS patch. If you find time, please have a 2.95.x compiler installed.

Another thing that could fail is if you additionally set

CONFIG_ACPI_FAN=m

and compile with CFLAGS="-g -ggdb"

Bonjour Willy,

Quoted 8 lines

Quoted 2 lines

Willy, if you have time, could you check your non-i386 boxes with a 2.95.x compiled 2.4.x kernel, with IPVS enabled?

Yes, no problem, but you'll have to tell me what to test ! (a config or script will save me some time). I have a Sun Ultra60 (ultrasparc SMP) which matches your description. I just have a doubt about gcc-2.95 availability on this box, I know I have a 3.3.6, do you think that the problem is gcc-related (too strong optimization or de-inlining, etc) ?

At least following should be set, the rest you can leave to your gusto:

CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BUS=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_MMCONFIG=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SYSTEM=y

CONFIG_IP_VS=m
CONFIG_IP_VS_DEBUG=y
CONFIG_IP_VS_TAB_BITS=12
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_HPRIO=m
CONFIG_IP_VS_FTP=m

One issue is a possible C99'ism in the last IPVS patch. If you find time, please have a 2.95.x compiler installed.

Another thing that could fail is if you additionally set

CONFIG_ACPI_FAN=m

and compile with CFLAGS="-g -ggdb"

Quoted 1 line

Please keep us informed when you have more info.

I will, and I will get more details, as time permits. My beef with the IPVS code seems to be wrong, the code works as expected so far. I'm stress-testing it though until Sunday on a 4GB Dual P4 Xeon with HT combo.

Quoted 10 lines

CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BUS=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_MMCONFIG=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SYSTEM=y

But this is purely x86-related, I won't have it on sparc.

Quoted 16 lines

CONFIG_IP_VS=m
CONFIG_IP_VS_DEBUG=y
CONFIG_IP_VS_TAB_BITS=12
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_HPRIO=m
CONFIG_IP_VS_FTP=m

Quoted 2 lines

One issue is a possible C99'ism in the last IPVS patch. If you find time, please have a 2.95.x compiler installed.

You mean that it's a build issue ? I first thought that you got erroneous behaviour.

How could I stress it ? what ipvs config, what type of traffic ? I'm used to stress-test firewalls and load-balancers, but there is a wide choice of possibilities, and all cannot be explored in a short timeframe.

Quoted 11 lines

Quoted 10 lines

CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BUS=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_MMCONFIG=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SYSTEM=y

Quoted 1 line

But this is purely x86-related, I won't have it on sparc.

Indeed ;).

Quoted 16 lines

Quoted 16 lines

CONFIG_IP_VS=m
CONFIG_IP_VS_DEBUG=y
CONFIG_IP_VS_TAB_BITS=12
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_HPRIO=m
CONFIG_IP_VS_FTP=m

Quoted 5 lines

Quoted 2 lines

One issue is a possible C99'ism in the last IPVS patch. If you find time, please have a 2.95.x compiler installed.

You mean that it's a build issue ? I first thought that you got erroneous behaviour.

Yes, the erroneous stuff I'm tracking down and it looks like I've found it (actually, Julian Anastasov fixed it):

diff -ur v2.4.32-rc2/linux/net/ipv4/ipvs/ip_vs_core.c linux/net/ipv4/ipvs/ip_vs_core.c
--- v2.4.32-rc2/linux/net/ipv4/ipvs/ip_vs_core.c 2005-11-03
01:20:02.000000000 +0200
+++ linux/net/ipv4/ipvs/ip_vs_core.c 2005-11-03 01:22:36.347895544 +0200

(quoted diff) +3 -4

@@ -1111,11 +1111,10 @@
 		if (sysctl_ip_vs_expire_nodest_conn) {
 			/* try to expire the connection immediately */
 			ip_vs_conn_expire_now(cp);
-		} else {
-			/* don't restart its timer, and silently
-			   drop the packet. */
-			__ip_vs_conn_put(cp);
 		}
+		/* don't restart its timer, and silently
+		   drop the packet. */
+		__ip_vs_conn_put(cp);
 		return NF_DROP;
 	}
 

I will send a proper signed-off and acked-by patch against rc2 after some more stress testing. So, please hold off releasing until then. I'm done testing this piece of code by tomorrow noon (GMT+1).

What I wasn't sure is if the latest patches still compiled on 2.95.x gcc. That's the only thing I wanted you to test.

I cannot ask you to run fully fledged LVS tests, as this requires quite some setup time.

Quoted 3 lines

How could I stress it ? what ipvs config, what type of traffic ? I'm used to stress-test firewalls and load-balancers, but there is a wide choice of possibilities, and all cannot be explored in a short timeframe.

You would need to test IPVS on a SMP box using persistent setup and 0 port feature and the expire_nodest_conn proc-fs entry set to 1. Hit the LB with 100Mbit/s traffic balancing it on 2-3 RS and reload the configuration using ipvsadm, _but_ without rmmod'ing the ip_vs_* kernel modules. Set the persistency timeout low (60 secs) and the timeout_finwait to 10*HZ. You need 2 clients which connect over a Linux router to a LVS_DR setup, one needs to be router through and the other should be NAT'd on the Linux router using a NAT pool to simulate 100's of clients. This way you have the slashdot-hype and the AOL proxy boost hitting your LB and generating loaded persistency templates which will then hit the code in question, wenn the internal timer expires. You need

to grep for NONE in ipvsadm -L -n -c to get the template entries. You must stop the client connecting directly through the Linux router after you reloaded the LB setup and then you observe the persistent template created for this client until the timer expires. Then you start it again and with luck you should see the abberant behaviour of a missed __ip_vs_conn_put(cp) :). I am pretty sure you do not want to go through this setup. I have it here and I'm stress testing all possible combinations of this szenario.

Thanks for your help, Willy.

A bientôt,
Roberto Nibali, ratz

Quoted 3 lines

I will send a proper signed-off and acked-by patch against rc2 after some more stress testing. So, please hold off releasing until then. I'm done testing this piece of code by tomorrow noon (GMT+1).

OK, fine. I'll merge it into next -hf (probably within a few days). Please insist loudly if you consider it important to fix quickly because it's a real regression, as I don't want to have people wait for long if one hotfix causes trouble.

OK, if it was your only concern, then I can say that it compiles and runs on x86.

Quoted 11 lines

You would need to test IPVS on a SMP box using persistent setup and 0 port feature and the expire_nodest_conn proc-fs entry set to 1. Hit the LB with 100Mbit/s traffic balancing it on 2-3 RS and reload the configuration using ipvsadm, _but_ without rmmod'ing the ip_vs_* kernel modules. Set the persistency timeout low (60 secs) and the timeout_finwait to 10*HZ. You need 2 clients which connect over a Linux router to a LVS_DR setup, one needs to be router through and the other should be NAT'd on the Linux router using a NAT pool to simulate 100's of clients. This way you have the slashdot-hype and the AOL proxy boost hitting your LB and generating loaded persistency templates which will then hit the code in question, wenn the internal timer expires. You need

Quoted 8 lines

to grep for NONE in ipvsadm -L -n -c to get the template entries. You must stop the client connecting directly through the Linux router after you reloaded the LB setup and then you observe the persistent template created for this client until the timer expires. Then you start it again and with luck you should see the abberant behaviour of a missed __ip_vs_conn_put(cp) :). I am pretty sure you do not want to go through this setup. I have it here and I'm stress testing all possible combinations of this szenario.

Of course this is not the easiest setup just to chase a bug down. But with such an explanation, a good manual on IPVS, and a lot of spare time, it could be done if it was the only solution. But I'm not willing to spend so much time on this yet :-)