Skip to contentSkip to conversation

OCFS2: runtime tunable network idle timeout

PATCH New wengang wang 2 participants 4 messages Jun 8, 2009 – Jun 9, 2009 OCFS2

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 OCFS2: runtime tunable network idle timeout

backgroud:
there is a network idle timeout regarding which a node is considered dead or network partition occures.

problem:
for some product environment, there is a special time during a day. in this special time, a backup work is happening over private network. at the time that the backup is going on, there is very very high load on network. this can lead to ocfs2 network idle timeout and when it can't connect back in time, some nodes have to be fensed out the cluster domain which is not really what we want. there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can set the timeout value. but looks it takes effect on when o2cb service is restarted, so it's not possible to change it in the already running system.

suggestion:
if we can modify the timeout value at runtime, it's better. we can add a proc file under /proc/fs/ocfs2_nodemanager, for example, idle_timeout, so that a userspace application(such as debugfs.ocfs2) can read/write the timeout value. before the customer run the backup, set the value to a big value(or to no limit) and set it back when backup finished. contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the timeout value in MS. 0 means no limit.

if it's good, I'm glad to do it.

thanks,
wengang.

#1 lore raw

PATCH 1/1 OCFS2: runtime tunable network idle timeout

Quoted 5 lines · wengang wang

backgroud:
there is a network idle timeout regarding which a node is considered dead or network partition occures.

problem:
for some product environment, there is a special time during a day. in this special time, a backup work is happening over private network. at the time that the backup is going on, there is very very high load on network. this can lead to ocfs2 network idle timeout and when it can't connect back in time, some nodes have to be fensed out the cluster domain which is not really what we want.

Bug#? SR? Have we ruled out a bug in our code? The last time I saw one of these
we determined it was because of a bug.

Quoted 7 lines

there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can set the timeout value. but looks it takes effect on when o2cb service is restarted, so it's not possible to change it in the already running system.

suggestion:
if we can modify the timeout value at runtime, it's better. we can add a proc file under /proc/fs/ocfs2_nodemanager, for example, idle_timeout, so that a userspace application(such as debugfs.ocfs2) can read/write the timeout value. before the customer run the backup, set the value to a big value(or to no limit) and set it back when backup finished. contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the timeout value in MS. 0 means no limit.

if it's good, I'm glad to do it.

One cannot just set this value on one node. It would have to be set atomically
on all nodes.

While that can still be done, my issue is as to why one cannot set that timeout
up front. Asking clients to "set" timeout dynamically before certain fs operations
is not at all friendly. Especially when the user has no idea as what workload a
certain operation entails.

#2 lore raw

PATCH 1/1 OCFS2: runtime tunable network idle timeout

Sunil,

Quoted 12 lines · Sunil Mushran
Quoted 10 lines · wengang wang

backgroud:
there is a network idle timeout regarding which a node is
considered dead or network partition occures.
problem:
for some product environment, there is a special time during a
day. in this special time, a backup work is happening over private network. at the time that the backup is going on, there is very very high load on network. this can lead to ocfs2 network idle timeout and when it can't connect back in time, some nodes have to be fensed out the cluster domain which is not really what we want.

Quoted 3 lines

Bug#? SR? Have we ruled out a bug in our code? The last time I saw one of these
we determined it was because of a bug.

one of the bugs is:
https://bug.oraclecorp.com/pls/bug/webbug_print.show?c_rptno=8443612

oh, sorry that I didn't notice it could be caused by a bug. will get tcpdumps to do more analyse on it..

Quoted 16 lines
Quoted 16 lines

there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can set
the timeout value. but looks it takes effect on when o2cb service is restarted, so it's not possible to change it in the already running system.

suggestion:
if we can modify the timeout value at runtime, it's better. we can
add a proc file under /proc/fs/ocfs2_nodemanager, for example, idle_timeout, so that a userspace application(such as debugfs.ocfs2) can read/write the timeout value. before the customer run the backup, set the value to a big value(or to no limit) and set it back when backup finished.
contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the timeout
value in MS. 0 means no limit.

Quoted 6 lines
Quoted 1 line

if it's good, I'm glad to do it.

One cannot just set this value on one node. It would have to be set atomically
on all nodes.

Yes, I know that.

Quoted 7 lines

While that can still be done, my issue is as to why one cannot set that timeout
up front. Asking clients to "set" timeout dynamically before certain fs operations
is not at all friendly. Especially when the user has no idea as what workload a
certain operation entails.

if the timeout is set as a too large value, I think it will cause slower response when a timeout happens(a true node death or network partition) for a normal network load. for a production environment, it's not good.

and yes it's difficult for clients to determine a high network load unless they has a very cool administrator -- that's a problem.

Ok, then we put it away now and put it up when we know clearly about the problem.

thanks
wengang.

Signature · 1 line
--just begin to learn, you are never too late...
#3 lore raw

PATCH 1/1 OCFS2: runtime tunable network idle timeout

This is the same dlm hash too small in 1.2. It has been addressed in 1.4.

Suggest client upgrade to 1.4.

Quoted 72 lines · Wengang Wang

Sunil,

Quoted 12 lines · Sunil Mushran
Quoted 10 lines · wengang wang

backgroud:
there is a network idle timeout regarding which a node is
considered dead or network partition occures.
problem:
for some product environment, there is a special time during a
day. in this special time, a backup work is happening over private network. at the time that the backup is going on, there is very very high load on network. this can lead to ocfs2 network idle timeout and when it can't connect back in time, some nodes have to be fensed out the cluster domain which is not really what we want.

Quoted 3 lines

Bug#? SR? Have we ruled out a bug in our code? The last time I saw one of these
we determined it was because of a bug.

one of the bugs is:
https://bug.oraclecorp.com/pls/bug/webbug_print.show?c_rptno=8443612

oh, sorry that I didn't notice it could be caused by a bug. will get tcpdumps to do more analyse on it..

Quoted 16 lines
Quoted 16 lines

there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can
set the timeout value. but looks it takes effect on when o2cb service is restarted, so it's not possible to change it in the already running system.

suggestion:
if we can modify the timeout value at runtime, it's better. we
can add a proc file under /proc/fs/ocfs2_nodemanager, for example, idle_timeout, so that a userspace application(such as debugfs.ocfs2) can read/write the timeout value. before the customer run the backup, set the value to a big value(or to no limit) and set it back when backup finished.
contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the
timeout value in MS. 0 means no limit.

Quoted 6 lines
Quoted 1 line

if it's good, I'm glad to do it.

One cannot just set this value on one node. It would have to be set atomically
on all nodes.

Yes, I know that.

Quoted 7 lines

While that can still be done, my issue is as to why one cannot set that timeout
up front. Asking clients to "set" timeout dynamically before certain fs operations
is not at all friendly. Especially when the user has no idea as what workload a
certain operation entails.

if the timeout is set as a too large value, I think it will cause slower response when a timeout happens(a true node death or network partition) for a normal network load. for a production environment, it's not good.

and yes it's difficult for clients to determine a high network load unless they has a very cool administrator -- that's a problem.

Ok, then we put it away now and put it up when we know clearly about the problem.

thanks
wengang.