Skip to contentSkip to conversation

ipvs_syncmaster brings cpu to 100%

DISCUSSION Horms 4 participants 13 messages Sep 26, 2005 – Sep 30, 2005 8 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Horms

author
#4 lore raw
Quoted 25 lines · Roger Tsang

As I've said before in this thread, you might want to try changing all the ssleep() calls to schedule_timeout().

Roger

Quoted 18 lines · Luca Maranzano

Hello all,

here again trying to discover the reason ot the CPU hog for ipvs_sync{master,backup}.

I've digged in the sources for ip_vs_sync.c and the main differences between kernel 2.6.8 and 2.6.12 is the use of ssleep() instead of schedule_timeout().

The oddity I've seen is that in the header of both files, the version is always like this:

* Version: $Id: ip_vs_sync.c,v 1.13 2003/06/08 09:31:19 wensong Exp $ *
* Authors: Wensong Zhang <[email protected]>

Is Wensong still the maintainer for this code?

Yes, although he is kind of quiet.

Quoted 8 lines
Quoted 8 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

I have CCed him, Dave Miller, Wensong Zhang, Julian Anastasov, and the netdev list for comment.

Could intererested parties please test the patch.

Thanks

Signature · 12 lines
Horms

Use schedule_timeout() instead of ssleep() in ip_vs_sync daemon,
as the latter seems to cause 100% CPU utilistaion on HT Xeons.

Discussion:
http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html

Reverts:
http://www.kernel.org/git/?p=linux/kernel/git/tglx/history.git;a=commit;h=f8afb60c7537130448cc479d6d8dc9bf4ee06027

Signed-off-by: Horms <[email protected]>
net/ipv4/ipvs/ip_vs_sync.c +12 -4
@@ -655,7 +655,9 @@ static void sync_master_loop(void)
 		if (stop_master_sync)
 			break;
 
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
+		__set_current_state(TASK_RUNNING);
 	}
 
 	/* clean up the sync_buff queue */
@@ -712,7 +714,9 @@ static void sync_backup_loop(void)
 		if (stop_backup_sync)
 			break;
 
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
+		__set_current_state(TASK_RUNNING);
 	}
 
 	/* release the sending multicast socket */
@@ -824,7 +828,9 @@ static int fork_sync_thread(void *startu
 	if ((pid = kernel_thread(sync_thread, startup, 0)) < 0) {
 		IP_VS_ERR("could not create sync_thread due to %d... "
 			  "retrying.\n", pid);
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
+		__set_current_state(TASK_RUNNING);
 		goto repeat;
 	}
 
@@ -858,7 +864,9 @@ int start_sync_thread(int state, char *m
 	if ((pid = kernel_thread(fork_sync_thread, &startup, 0)) < 0) {
 		IP_VS_ERR("could not create fork_sync_thread due to %d... "
 			  "retrying.\n", pid);
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
+		__set_current_state(TASK_RUNNING);
 		goto repeat;
 	}
 

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 33 lines · Horms
Quoted 25 lines · Roger Tsang

As I've said before in this thread, you might want to try changing all the ssleep() calls to schedule_timeout().

Roger

Quoted 18 lines · Luca Maranzano

Hello all,

here again trying to discover the reason ot the CPU hog for ipvs_sync{master,backup}.

I've digged in the sources for ip_vs_sync.c and the main differences between kernel 2.6.8 and 2.6.12 is the use of ssleep() instead of schedule_timeout().

The oddity I've seen is that in the header of both files, the version is always like this:

* Version: $Id: ip_vs_sync.c,v 1.13 2003/06/08 09:31:19 wensong Exp $ *
* Authors: Wensong Zhang <[email protected]>

Is Wensong still the maintainer for this code?

Yes, although he is kind of quiet.

Quoted 4 lines
Quoted 4 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

In your patch, you do not need to set the state back to TASK_RUNNING, btw.

Thanks,
Nish

author
#7 lore raw
Quoted 37 lines · Nishanth Aravamudan
Quoted 36 lines · Horms
Quoted 25 lines · Roger Tsang

As I've said before in this thread, you might want to try changing all the ssleep() calls to schedule_timeout().

Roger

Quoted 18 lines · Luca Maranzano

Hello all,

here again trying to discover the reason ot the CPU hog for ipvs_sync{master,backup}.

I've digged in the sources for ip_vs_sync.c and the main differences between kernel 2.6.8 and 2.6.12 is the use of ssleep() instead of schedule_timeout().

The oddity I've seen is that in the header of both files, the version is always like this:

* Version: $Id: ip_vs_sync.c,v 1.13 2003/06/08 09:31:19 wensong Exp $ *
* Authors: Wensong Zhang <[email protected]>

Is Wensong still the maintainer for this code?

Yes, although he is kind of quiet.

Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Quoted 2 lines

In your patch, you do not need to set the state back to TASK_RUNNING, btw.

Thanks, updated patch below.

Signature · 12 lines
Horms

Use schedule_timeout() instead of ssleep() in ip_vs_sync daemon,
as the latter seems to cause 100% CPU utilistaion on HT Xeons.

Discussion:
http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html

Reverts:
http://www.kernel.org/git/?p=linux/kernel/git/tglx/history.git;a=commit;h=f8afb60c7537130448cc479d6d8dc9bf4ee06027

Signed-off-by: Horms <[email protected]>
net/ipv4/ipvs/ip_vs_sync.c +8 -4
@@ -655,7 +655,8 @@ static void sync_master_loop(void)
 		if (stop_master_sync)
 			break;
 
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
 	}
 
 	/* clean up the sync_buff queue */
@@ -712,7 +713,8 @@ static void sync_backup_loop(void)
 		if (stop_backup_sync)
 			break;
 
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
 	}
 
 	/* release the sending multicast socket */
@@ -824,7 +826,8 @@ static int fork_sync_thread(void *startu
 	if ((pid = kernel_thread(sync_thread, startup, 0)) < 0) {
 		IP_VS_ERR("could not create sync_thread due to %d... "
 			  "retrying.\n", pid);
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
 		goto repeat;
 	}
 
@@ -858,7 +861,8 @@ int start_sync_thread(int state, char *m
 	if ((pid = kernel_thread(fork_sync_thread, &startup, 0)) < 0) {
 		IP_VS_ERR("could not create fork_sync_thread due to %d... "
 			  "retrying.\n", pid);
-		ssleep(1);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(HZ);
 		goto repeat;
 	}
 

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#9 lore raw

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

Quoted 8 lines

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Luca, please test.

Signature · 14 lines
Horms

*UNTESTED*

Use msleep_interruptible() instead of ssleep() in ip_vs_sync daemon,
as the latter seems to cause 100% CPU utilistaion on HT Xeons.

Discussion:
http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html

Reverts:
http://www.kernel.org/git/?p=linux/kernel/git/tglx/history.git;a=commit;h=f8afb60c7537130448cc479d6d8dc9bf4ee06027

Signed-off-by: Horms <[email protected]>
net/ipv4/ipvs/ip_vs_sync.c +4 -4
@@ -655,7 +655,7 @@ static void sync_master_loop(void)
 		if (stop_master_sync)
 			break;
 
-		ssleep(1);
+		msleep_interruptible(1000);
 	}
 
 	/* clean up the sync_buff queue */
@@ -712,7 +712,7 @@ static void sync_backup_loop(void)
 		if (stop_backup_sync)
 			break;
 
-		ssleep(1);
+		msleep_interruptible(1000);
 	}
 
 	/* release the sending multicast socket */
@@ -824,7 +824,7 @@ static int fork_sync_thread(void *startu
 	if ((pid = kernel_thread(sync_thread, startup, 0)) < 0) {
 		IP_VS_ERR("could not create sync_thread due to %d... "
 			  "retrying.\n", pid);
-		ssleep(1);
+		msleep_interruptible(1000);
 		goto repeat;
 	}
 
@@ -858,7 +858,7 @@ int start_sync_thread(int state, char *m
 	if ((pid = kernel_thread(fork_sync_thread, &startup, 0)) < 0) {
 		IP_VS_ERR("could not create fork_sync_thread due to %d... "
 			  "retrying.\n", pid);
-		ssleep(1);
+		msleep_interruptible(1000);
 		goto repeat;
 	}
 

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw
Quoted 28 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

Quoted 6 lines
Quoted 2 lines

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

Thanks,
Nish

#12 lore raw

Just to add more info, please note the output of "ps":

debld1:~# ps aux|grep ipvs
root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncmaster]
root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncbackup]

Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep (usually IO)

I hope to have a Xeon machine to make some more tests in the next days, in the mean time I'll try to reproduce my setup on a couple of VMWare Workstation machines.

More later.
Thank you all.
Luca

Quoted 45 lines · Nishanth Aravamudan
Quoted 28 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

Quoted 6 lines
Quoted 2 lines

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

Thanks,
Nish

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw
Quoted 37 lines · Luca Maranzano
Quoted 36 lines · Nishanth Aravamudan
Quoted 34 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

Quoted 8 lines

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Quoted 5 lines
Quoted 5 lines

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

[trimmed lvs-users from my reply, as it is a closed list]

Quoted 10 lines

Just to add more info, please note the output of "ps":

debld1:~# ps aux|grep ipvs
root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncmaster]
root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncbackup]

Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep (usually IO)

The msleep_interruptible() change should fix that.

But that does not show 100% CPU utilisation at all, it shows 0. Did you mean to say your load increases?

I'm still unclear what the problem is. Horms initial Cc trimmed some important information. It would be very useful to "start over" -- at least from the perspective of what the problem actually is.

Quoted 3 lines

I hope to have a Xeon machine to make some more tests in the next days, in the mean time I'll try to reproduce my setup on a couple of VMWare Workstation machines.

Please don't top-most. It makes it really hard to write sane replies...

Thanks,
Nish

#15 lore raw

[trimmed Cc to avoid spamming...]

Ok, just to summarize the long thread from the beginning:

The goal: setting up a Local Director with IPVS with state synchronization, failover and failback.

The hardware: 1 CPU Intel Xeon 3,4 Ghz - HP DL380G4 on 2 identical boxes

The problems (please note that all kernel versions are *Debian* kernels):
1. Kernel 2.6.8: got a system lock of the standby node when simulating a failover. The load average as reported from "top" or "w" is always 0.00.

2. Kernel 2.6.11 and Kernel 2.6.12: failover and failback works fine, but the load average as reported from "top" or "w" is always systematically at 2.00 or more with both sync thread started (ipvs_syncmaster and ipvs_syncbackup). Load average from top is 1.00 or mroe with only one thread (i.e. ipvs_syncmaster). Horms reported that he was not able to reproduce this on a non-Xeon system.

That's all, let me know if you need more info.
Regards,
Luca

Quoted 75 lines · Nishanth Aravamudan
Quoted 37 lines · Luca Maranzano
Quoted 36 lines · Nishanth Aravamudan
Quoted 34 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

Quoted 8 lines

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Quoted 5 lines
Quoted 5 lines

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

[trimmed lvs-users from my reply, as it is a closed list]

Quoted 10 lines

Just to add more info, please note the output of "ps":

debld1:~# ps aux|grep ipvs
root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncmaster]
root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncbackup]

Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep (usually IO)

The msleep_interruptible() change should fix that.

But that does not show 100% CPU utilisation at all, it shows 0. Did you mean to say your load increases?

I'm still unclear what the problem is. Horms initial Cc trimmed some important information. It would be very useful to "start over" -- at least from the perspective of what the problem actually is.

Quoted 3 lines

I hope to have a Xeon machine to make some more tests in the next days, in the mean time I'll try to reproduce my setup on a couple of VMWare Workstation machines.

Please don't top-most. It makes it really hard to write sane replies...

Thanks,
Nish

#16 lore raw
Quoted 87 lines · Luca Maranzano
Quoted 66 lines · Nishanth Aravamudan
Quoted 37 lines · Luca Maranzano
Quoted 36 lines · Nishanth Aravamudan
Quoted 34 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

Quoted 8 lines

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Quoted 5 lines
Quoted 5 lines

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

[trimmed lvs-users from my reply, as it is a closed list]

Quoted 10 lines

Just to add more info, please note the output of "ps":

debld1:~# ps aux|grep ipvs
root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncmaster]
root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncbackup]

Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep (usually IO)

The msleep_interruptible() change should fix that.

But that does not show 100% CPU utilisation at all, it shows 0. Did you mean to say your load increases?

I'm still unclear what the problem is. Horms initial Cc trimmed some important information. It would be very useful to "start over" -- at least from the perspective of what the problem actually is.

Quoted 5 lines
Quoted 3 lines

I hope to have a Xeon machine to make some more tests in the next days, in the mean time I'll try to reproduce my setup on a couple of VMWare Workstation machines.

Please don't top-most. It makes it really hard to write sane replies...

[trimmed Cc to avoid spamming...]

Ok, just to summarize the long thread from the beginning:

The goal: setting up a Local Director with IPVS with state synchronization, failover and failback.

The hardware: 1 CPU Intel Xeon 3,4 Ghz - HP DL380G4 on 2 identical boxes

The problems (please note that all kernel versions are *Debian* kernels):
1. Kernel 2.6.8: got a system lock of the standby node when simulating a failover. The load average as reported from "top" or "w" is always 0.00.

Quoted 6 lines

2. Kernel 2.6.11 and Kernel 2.6.12: failover and failback works fine, but the load average as reported from "top" or "w" is always systematically at 2.00 or more with both sync thread started (ipvs_syncmaster and ipvs_syncbackup). Load average from top is 1.00 or mroe with only one thread (i.e. ipvs_syncmaster). Horms reported that he was not able to reproduce this on a non-Xeon system.

Ok, so when whomever mentioned "CPU utilisation" they were mistaken. The load average being 2 is due to ssleep(). The msleep_interruptible() version of the patch should fix that up. It really doesn't make any difference in the code, except that your load average will go back to 0.00 and the ipvs threads can be interrupted by signals.

I would expect the load average to be 2.00 for all systems, not just Xeon. The system lock has nothing to do with the patch, though. Something else fixed it.

Thanks,
Nish

P.S. Again, please don't top-post, it makes it harder for me to reply (and disinclines me to do so).

author
#17 lore raw
In reply to Nishanth Aravamudan
Quoted 54 lines · Nishanth Aravamudan
Quoted 37 lines · Luca Maranzano
Quoted 36 lines · Nishanth Aravamudan
Quoted 34 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

Quoted 8 lines

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Quoted 5 lines
Quoted 5 lines

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

[trimmed lvs-users from my reply, as it is a closed list]

Quoted 8 lines

Just to add more info, please note the output of "ps":

debld1:~# ps aux|grep ipvs
root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncmaster]
root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncbackup]
Quoted 7 lines
Quoted 2 lines

Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep (usually IO)

The msleep_interruptible() change should fix that.

But that does not show 100% CPU utilisation at all, it shows 0. Did you mean to say your load increases?

he full discussion is available online at the follwoing URL:
I can get than information and post it all here if that is desirable.

http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html

Signature · 1 line
Horms
#18 lore raw
Quoted 60 lines · Horms
Quoted 59 lines · Nishanth Aravamudan
Quoted 37 lines · Luca Maranzano
Quoted 36 lines · Nishanth Aravamudan
Quoted 34 lines · Horms

On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:

[snip]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12 the function schedule_timeout() is more used than the ssleep() (517 occurrencies vs. 43), so why in ip_vs_sync.c there was this change?

The other oddity is that Horms reported on this list that on non Xeon CPU the same version of kernel of mine does not present the problem.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I'm getting crazy :-)

I've prepared a patch, which reverts the change which was introduced by Nishanth Aravamudan in February.

Was the 100% cpu utilization only occurring on Xeon processors?

That seems to be the only case where were this problem has been observed. I don't have such a processor myself, so I haven't actually been able to produce the problem locally.

Quoted 8 lines

One reason I posted this issue to netdev was to get some more eyes on the problem as it is puzzling to say the least.

Quoted 2 lines

Care to try to use msleep_interruptible() instead of ssleep(), as opposed to schedule_timeout()?

I will send a version that does that shortly, Luca, can you plase check that too?

Here is that version of the patch. Nishanth, I take it that I do not need to set TASK_INTERRUPTABLE before calling msleep_interruptible(), please let me know if I am wrong.

Quoted 5 lines
Quoted 5 lines

Yes, exactly. I'm just trying to narrow it down to see if it's the task state that's causing the issue (which, to be honest, doesn't make a lot of sense to me -- with ssleep() your load average will go up as the task will be UNINTERRUPTIBLE state, but I am not sure why utilisation would rise, as you are still sleeping...)

[trimmed lvs-users from my reply, as it is a closed list]

Quoted 10 lines

Just to add more info, please note the output of "ps":

debld1:~# ps aux|grep ipvs
root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncmaster]
root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
[ipvs_syncbackup]

Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep (usually IO)

The msleep_interruptible() change should fix that.

Quoted 8 lines
Quoted 2 lines

But that does not show 100% CPU utilisation at all, it shows 0. Did you mean to say your load increases?

he full discussion is available online at the follwoing URL:
I can get than information and post it all here if that is desirable.

http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html

Yes, the information in that thread is the same as what Luca said. It's a load average problem, not a CPU utilisation problem (those threads are sleeping!) If Luca could test the msleep_interruptible() version of the patch and it works (like I said, performance should not change, but the load average will drop to by 2), then I will ACK the patch for mainline acceptance.

Thanks,
Nish

#19 lore raw

Hello,

Quoted 6 lines · Nishanth Aravamudan

Yes, the information in that thread is the same as what Luca said. It's a load average problem, not a CPU utilisation problem (those threads are sleeping!) If Luca could test the msleep_interruptible() version of the patch and it works (like I said, performance should not change, but the load average will drop to by 2), then I will ACK the patch for mainline acceptance.

Agreed. It seems your initial conversion was based on wrong
assumptions, quoting you:

Quoted 4 lines

Description: Use ssleep() instead of schedule_timeout() to guarantee the task delays as expected. The first two replacements use TASK_INTERRUPTIBLE but do not
check for signals, so ssleep() should be appropriate.

As all signals are blocked from daemonize and even explicitly
later it was not necessary to convert to non-interruptible variant.

Regards

--
Julian Anastasov <[email protected]>

#20 lore raw
In reply to Nishanth Aravamudan

First of all thank you all for your precious support! :-)

The two machines on which I discovered the problem are now in production and I cannot for the moment make tests, but I hope to have some other hardware to try in the next week.
I'll let you know ASAP.

Thanks,
Luca

Quoted 11 lines

Yes, the information in that thread is the same as what Luca said. It's a load average problem, not a CPU utilisation problem (those threads are sleeping!) If Luca could test the msleep_interruptible() version of the patch and it works (like I said, performance should not change, but the load average will drop to by 2), then I will ACK the patch for mainline acceptance.

Thanks,
Nish