Skip to contentSkip to conversation

DoS with NFSv4.1 client

DISCUSSION Mkrtchyan, Tigran 3 participants 8 messages Oct 9, 2013 – Oct 10, 2013 4 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094 (milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time: 2057.365517 (milliseconds)

As you can see there is a quite a big difference between number of open and close requests. The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns) max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000 97000 17

  OPEN                                    1197297±  0.00       816000  31924558000          54398533
  RESTOREFH                                     0±  0.00            0  25018778000          54398533
  SEQUENCE                                   1000±  0.00         1000  26066722000          55601046
  LOOKUP                                  4607959±  0.00       375000  26977455000             32118
  GETDEVICEINFO                             13158±100.88         4000       655000             11378
  CLOSE                                  16236211±  0.00         5000  21021819000             20420
  LAYOUTGET                             271736361±  0.00     10003000  68414723000             21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.

#2 lore raw

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 78 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time: 2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw

So is this a server bug? It seems like the client is behaving correctly...

-dros

Quoted 101 lines · Tigran

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 69 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time: 2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

Quoted 9 lines

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

Well, it'd be nice not to loop forever, but my question remains, is this due to a server bug (the DS not knowing about new stateid from MDS)?

-dros

Quoted 112 lines · Weston Andros Adamson

So is this a server bug? It seems like the client is behaving correctly...

-dros

Quoted 97 lines · Tigran

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 69 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time: 2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

Quoted 9 lines

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw

----- Original Message -----

Quoted 10 lines

From: "Weston Andros Adamson" <[email protected]>
To: "Tigran Mkrtchyan" <[email protected]>
Cc: "<[email protected]>" <[email protected]>, "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Thursday, October 10, 2013 4:35:25 PM
Subject: Re: DoS with NFSv4.1 client

Well, it'd be nice not to loop forever, but my question remains, is this due to a server bug (the DS not knowing about new stateid from MDS)?

Up to now, we have pushed open state id to the DS only on LAYOUTGET. This have to be changed, as the behaviour is not spec compliant.

Tigran.

Quoted 122 lines

-dros

Quoted 116 lines · Weston Andros Adamson

So is this a server bug? It seems like the client is behaving correctly...

-dros

Quoted 100 lines · Tigran

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 72 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time:
	2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and
close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

Quoted 9 lines

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

This is probably a question to IEFT working group, but anyway. If my layout has a flag 'return-on-close' and open state id is not valid any more should client expect layout to be still valid?

Tigran.

----- Original Message -----

Quoted 158 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: "Weston Andros Adamson" <[email protected]>
Cc: "linux-nfs" <[email protected]>, "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Thursday, October 10, 2013 4:48:52 PM
Subject: Re: DoS with NFSv4.1 client

----- Original Message -----

Quoted 8 lines

From: "Weston Andros Adamson" <[email protected]>
To: "Tigran Mkrtchyan" <[email protected]>
Cc: "<[email protected]>" <[email protected]>, "Andy Adamson" <[email protected]>, "Steve
Dickson" <[email protected]>
Sent: Thursday, October 10, 2013 4:35:25 PM
Subject: Re: DoS with NFSv4.1 client

Quoted 4 lines

Well, it'd be nice not to loop forever, but my question remains, is this due
to a server bug (the DS not knowing about new stateid from MDS)?

Up to now, we have pushed open state id to the DS only on LAYOUTGET. This have to be changed, as the behaviour is not spec compliant.

Tigran.

Quoted 125 lines

-dros

Quoted 118 lines · Weston Andros Adamson

So is this a server bug? It seems like the client is behaving correctly...

-dros

Quoted 100 lines · Tigran

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 72 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time:
	2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and
close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

Quoted 9 lines

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 5 lines · Tigran

This is probably a question to IEFT working group, but anyway. If my layout has a flag 'return-on-close' and open state id is not valid any more should client expect layout to be still valid?

Here is my take:

The layout stateid is constructed from the first open stateid when pNFS I/O is tried on that file. Once the layout return is successful, the layout stateid is independent from the open stateid used to construct it. So if that open, or another open stateid goes bad, the layout stateid is still valid.

WRT return-on-close, the invalid openstateid means there is no CLOSE until after the OPEN stateid is recovered (CLAIM_PREVIOUS) and the CLOSE call has a valid stateid. No CLOSE on an invalid stateid means no return-on-close for the invalid stateid which means the layout is still valid until the CLOSE using the recovered open stateid.

-->Andy

Quoted 162 lines

Tigran.

----- Original Message -----

Quoted 158 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: "Weston Andros Adamson" <[email protected]>
Cc: "linux-nfs" <[email protected]>, "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Thursday, October 10, 2013 4:48:52 PM
Subject: Re: DoS with NFSv4.1 client

----- Original Message -----

Quoted 8 lines

From: "Weston Andros Adamson" <[email protected]>
To: "Tigran Mkrtchyan" <[email protected]>
Cc: "<[email protected]>" <[email protected]>, "Andy Adamson" <[email protected]>, "Steve
Dickson" <[email protected]>
Sent: Thursday, October 10, 2013 4:35:25 PM
Subject: Re: DoS with NFSv4.1 client

Quoted 4 lines

Well, it'd be nice not to loop forever, but my question remains, is this due
to a server bug (the DS not knowing about new stateid from MDS)?

Up to now, we have pushed open state id to the DS only on LAYOUTGET. This have to be changed, as the behavior is not spec compliant.

Tigran.

Quoted 125 lines

-dros

Quoted 118 lines · Weston Andros Adamson

So is this a server bug? It seems like the client is behaving correctly...

-dros

Quoted 100 lines · Tigran

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 72 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time:
	2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and
close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

Quoted 9 lines

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw

Sorry - I answered this email thread from my netapp account and didn't 'cc the lists.

-->Andy

Quoted 136 lines · Andy
Quoted 2 lines · Tigran

Not only. As was able to reproduce it and fix on the server, we see that at the end client will send only one CLOSE.

I don't understand. If it is fixed on the server, then the client will send an OPEN, get an openstateid - say OS-1 , do a LAYOUTGET, and READ to the DS using OS-1. The server then returns BAD stateid on the READ.

The client then goes through stateid recovery, which means issuing another OPEN to get OS-2, which is then used for the DS READS

the client then CLOSE the file using OS-2.

Are you saying that the client does not close using OS-1? Note that is impossible, as OS-1 is a BAD stateid….

-->Andy

Quoted 118 lines

Tigran.

----- Original Message -----

Quoted 114 lines

From: "Andy Adamson" <[email protected]>
To: "Tigran Mkrtchyan" <[email protected]>
Sent: Thursday, October 10, 2013 3:55:55 PM
Subject: Re: DoS with NFSv4.1 client

OK - so it's a server bug,

-->Andy

Quoted 100 lines · Tigran

Today we was 'luck' to have such situation at day time. Here is what happens:

The client sends an OPEN and gets an open state id. This is followed by LAYOUTGET ... and READ to DS. At some point, server returns back BAD_STATEID. This triggers client to issue a new OPEN and use new open stateid with READ request to DS. As new stateid is not known to DS, it keeps returning BAD_STATEID and becomes an infinite loop.

Regards,
Tigran.

----- Original Message -----

Quoted 81 lines

From: "Tigran Mkrtchyan" <[email protected]>
To: [email protected]
Cc: "Andy Adamson" <[email protected]>, "Steve Dickson" <[email protected]>
Sent: Wednesday, October 9, 2013 10:48:32 PM
Subject: DoS with NFSv4.1 client

Hi,

last night we got a DoS attack with one of the NFS clients. The farm node, which was accessing data with pNFS, went mad and have tried to kill dCache NFS server. As usually this have happened over night and we was not able to get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see state created on the server side, the requests was processed by server. Nevertheless, for some reason, client did not like it. Here is the result of mountstats:

OPEN:

	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094
	(milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time:
	2057.365517
	(milliseconds)

As you can see there is a quite a big difference between number of open and
close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats: average±stderr(ns) min(ns)
max(ns) Sampes
DESTROY_SESSION 26056±4511.89 13000
97000 17
OPEN 1197297± 0.00 816000
31924558000 54398533
RESTOREFH 0± 0.00 0
25018778000 54398533
SEQUENCE 1000± 0.00 1000
26066722000 55601046
LOOKUP 4607959± 0.00 375000
26977455000 32118
GETDEVICEINFO 13158±100.88 4000
655000 11378
CLOSE 16236211± 0.00 5000
21021819000 20420
LAYOUTGET 271736361± 0.00 10003000
68414723000 21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code, I can see a loop at nfs4proc.c#nfs4_do_open() which can be the cause of the problem. Nevertheless, I can't fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned NFSERR_SERVERFAULT to the client. This triggered client to reestablish the session and all open state ids was invalidated and cleaned up.

I am still trying to reproduce this behavior (on client and server) and any hint is welcome.

Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html