pNFS: invalid IP:port selection when talks to DS

DISCUSSION Mkrtchyan, Tigran 2 participants 7 messages Mar 20, 2017 – Mar 22, 2017

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Mkrtchyan, Tigran

author

2017-03-20 16:35 UTC #0 lore raw

Dear (p)NFS-ors,

we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:

If client uses different DSes on the same host, then at some point it starts to send data to the wrong port number:

Client <=> MDS:

    1 0.000000000 131.169.251.53 → 131.169.51.35 NFS V4 Call OPEN DH: 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
    2 0.001469799 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 1) OPEN StateID: 0xec18
    3 0.001578128 131.169.251.53 → 131.169.51.35 NFS V4 Call SETATTR FH: 0x6ccf3dfa
    4 0.002657187 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 3) SETATTR
    5 0.003243819 131.169.251.53 → 131.169.51.35 NFS V4 Call LAYOUTGET
    6 0.014603386 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 5) LAYOUTGET
    7 0.014899121 131.169.251.53 → 131.169.51.35 NFS V4 Call GETDEVINFO
    8 0.015014216 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 7) GETDEVINFO
        Opcode: GETDEVINFO (47)
            Status: NFS4_OK (0)
            layout type: LAYOUT4_NFSV4_1_FILES (1)
            device index: 0
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            notification bitmap: 6
            notification bitmap: 0
    [Main Opcode: GETDEVINFO (47)]

    9 0.105442455 131.169.251.53 → 131.169.51.35 NFS V4 Call TEST_STATEID

10 0.105521354 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 9) TEST_STATEID

NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.

client <=> DS

$ tshark -r ds-write.pcap  -n -z conv,tcp
    1   0.000000 131.169.251.53 → 131.169.51.50 NFS V4 Call WRITE StateID: 0xff01 Offset: 0 Len: 3968
    2   0.000090 131.169.51.50 → 131.169.251.53 NFS V4 Reply (Call In 1) WRITE Status: NFS4ERR_BAD_STATEID
================================================================================
TCP Conversations
Filter:<No Filter>
                                                           |       <-      | |       ->      | |     Total     |    Relative    |   Duration   |
                                                           | Frames  Bytes | | Frames  Bytes | | Frames  Bytes |      Start     |              |
131.169.51.50:24006        <-> 131.169.251.53:847               1      4240       1       168       2      4408     0.000000000         0.0001
================================================================================

NOTICE, that it talks to DS on port 24006!

Is there know fix which is missing in CentOS7? I can't reproduce it with 4.9 kernel (or it's harder to reproduce).

The packages are attached.

Tigran.

Mkrtchyan, Tigran

author

2017-03-20 16:16 UTC #1 lore raw

re-sending without attachments.
The capture failes can be found at:

client <-> mds: https://desycloud.desy.de/index.php/s/58JFyfMQmNF99pU
client <-> ds: https://desycloud.desy.de/index.php/s/dKf290ikQcifL9K

Tigran.

----- Original Message -----

Quoted 6 lines

From: "Mkrtchyan, Tigran" <[email protected]>
To: "Linux NFS Mailing list" <[email protected]>
Cc: "Steve Dickson" <[email protected]>
Sent: Monday, March 20, 2017 4:52:40 PM
Subject: pNFS: invalid IP:port selection when talks to DS

Quoted 8 lines

Dear (p)NFS-ors,
=20
we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:
=20
If client uses different DSes on the same host, then at some point it sta=

rts

Quoted 6 lines

to send data to the wrong port number:
=20
Client <=3D> MDS:
=20
=20
1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OPEN =

DH:

Quoted 2 lines

   0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
   2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=

l In 1) OPEN

Quoted 2 lines

   StateID: 0xec18
   3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SETAT=

TR FH: 0x6ccf3dfa

Quoted 1 line

   4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=

l In 3) SETATTR

Quoted 1 line

   5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LAYOU=

TGET

Quoted 1 line

   6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=

l In 5) LAYOUTGET

Quoted 1 line

   7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GETDE=

VINFO

Quoted 1 line

   8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=

l In 7) GETDEVINFO

Quoted 20 lines

       Opcode: GETDEVINFO (47)
           Status: NFS4_OK (0)
           layout type: LAYOUT4_NFSV4_1_FILES (1)
           device index: 0
           r_netid: tcp
               length: 3
               contents: tcp
               fill bytes: opaque data
           r_addr: 131.169.51.50.93.197
               length: 20
               contents: 131.169.51.50.93.197
           r_netid: tcp
               length: 3
               contents: tcp
               fill bytes: opaque data
           r_addr: 131.169.51.50.93.197
               length: 20
               contents: 131.169.51.50.93.197
           notification bitmap: 6
           notification bitmap: 0

Quoted 3 lines

[Main Opcode: GETDEVINFO (47)]
=20
9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TEST_=

STATEID

Quoted 1 line

10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=

l In 9)

Quoted 10 lines

TEST_STATEID
=20
=20
=20
NOTICE, that 131.169.51.50.93.197 corresponds to port 24005. =20
client <=3D> DS
=20
$ tshark -r ds-write.pcap  -n -z conv,tcp
   1   0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRITE =

StateID: 0xff01

Quoted 2 lines

   Offset: 0 Len: 3968
   2   0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (Call=

In 1) WRITE

Quoted 2 lines

Status: NFS4ERR_BAD_STATEID
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Quoted 3 lines

TCP Conversations
Filter:<No Filter>
| <- =

 | |       ->      | |     Total     |    Relative    |   Duration   |
>                                                           | Frames  Bytes=
 | | Frames  Bytes | | Frames  Bytes |      Start     |
>                                                           | |
> 131.169.51.50:24006        <-> 131.169.251.53:847               1      42=

Quoted 2 lines

1 168 2 4408 0.000000000 0.0001
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Quoted 10 lines

=20
NOTICE, that it talks to DS on port 24006!
=20
Is there know fix which is missing in CentOS7? I can't reproduce it with 4.9 kernel (or it's harder to reproduce).
=20
=20
The packages are attached.
=20
Tigran.

Olga Kornievskaia

2017-03-20 20:14 UTC #2 lore raw

In reply to Mkrtchyan, Tigran

Hi Tigran,

While I don't have an answer to your question, I'd like to point out that in 4.9 is when Andy's session trunking patches when in.

I'm curious this client that's now talking to the DS at port 24006 instead of 24005, did it before also earlier correctly (legally) talked to DS that was on 24006?

Quoted 10 lines · Tigran

Dear (p)NFS-ors,
we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:
If client uses different DSes on the same host, then at some point it sta=

rts

Quoted 6 lines

to send data to the wrong port number:
Client <=3D> MDS:
    1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OPEN=

DH: 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata

Quoted 1 line

    2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=

ll In 1) OPEN StateID: 0xec18

Quoted 1 line

    3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SETA=

TTR FH: 0x6ccf3dfa

Quoted 1 line

    4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=

ll In 3) SETATTR

Quoted 1 line

    5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LAYO=

UTGET

Quoted 1 line

    6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=

ll In 5) LAYOUTGET

Quoted 1 line

    7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GETD=

EVINFO

Quoted 1 line

    8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=

ll In 7) GETDEVINFO

Quoted 15 lines

        Opcode: GETDEVINFO (47)
            Status: NFS4_OK (0)
            layout type: LAYOUT4_NFSV4_1_FILES (1)
            device index: 0
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data

Quoted 8 lines

            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            notification bitmap: 6
            notification bitmap: 0
    [Main Opcode: GETDEVINFO (47)]

    9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TEST=

_STATEID

Quoted 1 line

   10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=

ll In 9) TEST_STATEID

Quoted 9 lines

NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
client <=3D> DS
$ tshark -r ds-write.pcap  -n -z conv,tcp
    1   0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRITE=

StateID: 0xff01 Offset: 0 Len: 3968

Quoted 1 line

    2   0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=

l In 1) WRITE Status: NFS4ERR_BAD_STATEID

Quoted 1 line

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Quoted 3 lines

TCP Conversations
Filter:<No Filter>

                                                           |       <-    =

  | |       ->      | |     Total     |    Relative    |   Duration   |
>                                                            | Frames  Byte=
s | | Frames  Bytes | | Frames  Bytes |      Start     |              |
> 131.169.51.50:24006        <-> 131.169.251.53:847               1      42=
40       1       168       2      4408     0.000000000         0.0001

Quoted 1 line

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Quoted 11 lines

NOTICE, that it talks to DS on port 24006!
Is there know fix which is missing in CentOS7? I can't reproduce it with 4.9 kernel (or it's harder to reproduce).
The packages are attached.
Tigran.

Mkrtchyan, Tigran

author

2017-03-20 20:51 UTC #3 lore raw

Hi Olga,

----- Original Message -----

Quoted 3 lines

From: "Olga Kornievskaia" <[email protected]>
To: "Mkrtchyan, Tigran" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickson"=

<[email protected]>

Quoted 11 lines

Sent: Monday, March 20, 2017 9:14:34 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Hi Tigran,
=20
While I don't have an answer to your question, I'd like to point out that in 4.9 is when Andy's session trunking patches when in. =20
I'm curious this client that's now talking to the DS at port 24006 instead of 24005, did it before also earlier correctly (legally) talked to DS that was on 24006?

Yes, earlier during testing it had legal access to DS on port 24006.

Tigran.

Quoted 13 lines

=20
Quoted 10 lines · Tigran
Dear (p)NFS-ors,
we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:
If client uses different DSes on the same host, then at some point it st=

arts

Quoted 6 lines

Quoted 6 lines
to send data to the wrong port number:
Client <=3D> MDS:
    1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OPE=

N DH:

Quoted 2 lines

    0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
    2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=

all In 1) OPEN

Quoted 2 lines

    StateID: 0xec18
    3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SET=

ATTR FH: 0x6ccf3dfa

Quoted 1 line

    4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=

all In 3) SETATTR

Quoted 1 line

    5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LAY=

OUTGET

Quoted 1 line

    6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=

all In 5) LAYOUTGET

Quoted 1 line

    7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GET=

DEVINFO

Quoted 1 line

    8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=

all In 7) GETDEVINFO

Quoted 22 lines

        Opcode: GETDEVINFO (47)
            Status: NFS4_OK (0)
            layout type: LAYOUT4_NFSV4_1_FILES (1)
            device index: 0
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            notification bitmap: 6
            notification bitmap: 0
    [Main Opcode: GETDEVINFO (47)]

Quoted 1 line

    9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TES=

T_STATEID

Quoted 1 line

   10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=

all In 9)

Quoted 10 lines

Quoted 10 lines
TEST_STATEID
NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
client <=3D> DS
$ tshark -r ds-write.pcap  -n -z conv,tcp
    1   0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRIT=

E StateID: 0xff01

Quoted 2 lines

    Offset: 0 Len: 3968
    2   0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=

ll In 1) WRITE

Quoted 2 lines

Quoted 2 lines
Status: NFS4ERR_BAD_STATEID
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Quoted 3 lines

TCP Conversations
Filter:<No Filter>

                                                           |       <-   =

   | |       ->      | |     Total     |    Relative    |   Duration   |
>>                                                            | Frames  Byt=
es | | Frames  Bytes | | Frames  Bytes |      Start     |
>>                                                            | |
>> 131.169.51.50:24006        <-> 131.169.251.53:847               1      4=

240

Quoted 2 lines

Quoted 2 lines
1 168 2 4408 0.000000000 0.0001
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Quoted 15 lines

Quoted 11 lines
NOTICE, that it talks to DS on port 24006!
Is there know fix which is missing in CentOS7? I can't reproduce it with 4.9 kernel (or it's harder to reproduce).
The packages are attached.
Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Mkrtchyan, Tigran

author

2017-03-20 21:10 UTC #4 lore raw

Hi Olga,

you did not have the answer, however you gave me an important hint! I believe, all our DSes on a single host generate the same server owner during exchange-id. I guess, this can be the reason, why client decides to talk to an other DS.

Tigran.

----- Original Message -----

Quoted 3 lines

From: "Mkrtchyan, Tigran" <[email protected]>
To: "Olga Kornievskaia" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickson"=

<[email protected]>

Quoted 9 lines

Sent: Monday, March 20, 2017 9:51:21 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Hi Olga,
=20
----- Original Message -----
Quoted 3 lines
From: "Olga Kornievskaia" <[email protected]>
To: "Mkrtchyan, Tigran" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickson=

Quoted 12 lines

Quoted 3 lines
<[email protected]>
Sent: Monday, March 20, 2017 9:14:34 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
=20
Quoted 8 lines
Hi Tigran,
=20
While I don't have an answer to your question, I'd like to point out that in 4.9 is when Andy's session trunking patches when in. =20
I'm curious this client that's now talking to the DS at port 24006 instead of 24005, did it before also earlier correctly (legally) talked to DS that was on 24006?

Quoted 18 lines

=20
Yes, earlier during testing it had legal access to DS on port 24006. =20
Tigran.
=20
Quoted 13 lines
=20
Quoted 10 lines · Tigran
Dear (p)NFS-ors,
we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:
If client uses different DSes on the same host, then at some point it s=

tarts

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
to send data to the wrong port number:
Client <=3D> MDS:
    1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OP=

EN DH:

Quoted 2 lines

    0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
    2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=

Call In 1) OPEN

Quoted 2 lines

    StateID: 0xec18
    3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SE=

TATTR FH: 0x6ccf3dfa

Quoted 1 line

    4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=

Call In 3) SETATTR

Quoted 1 line

    5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LA=

YOUTGET

Quoted 1 line

    6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=

Call In 5) LAYOUTGET

Quoted 1 line

    7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GE=

TDEVINFO

Quoted 1 line

    8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=

Call In 7) GETDEVINFO

Quoted 22 lines

        Opcode: GETDEVINFO (47)
            Status: NFS4_OK (0)
            layout type: LAYOUT4_NFSV4_1_FILES (1)
            device index: 0
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            notification bitmap: 6
            notification bitmap: 0
    [Main Opcode: GETDEVINFO (47)]

Quoted 1 line

    9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TE=

ST_STATEID

Quoted 1 line

   10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=

Call In 9)

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
TEST_STATEID
NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
client <=3D> DS
$ tshark -r ds-write.pcap  -n -z conv,tcp
    1   0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRI=

TE StateID: 0xff01

Quoted 2 lines

    Offset: 0 Len: 3968
    2   0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (C=

all In 1) WRITE

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Status: NFS4ERR_BAD_STATEID
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Quoted 3 lines

TCP Conversations
Filter:<No Filter>

                                                           |       <-  =

    | |       ->      | |     Total     |    Relative    |   Duration   |
>>>                                                            | Frames  By=
tes | | Frames  Bytes | | Frames  Bytes |      Start     |
>>>                                                            | |
>>> 131.169.51.50:24006        <-> 131.169.251.53:847               1      =

4240

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
1 168 2 4408 0.000000000 0.0001
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
NOTICE, that it talks to DS on port 24006!
Is there know fix which is missing in CentOS7? I can't reproduce it wit=

Quoted 15 lines

Quoted 11 lines
Quoted 7 lines
4.9 kernel (or it's harder to reproduce).
The packages are attached.
Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Olga Kornievskaia

2017-03-22 16:11 UTC #5 lore raw

Hi Tigran,

I still don't have the answer to your question but I'm just puzzled why it "works" with 4.9 (session trunking). New code would check the server owner and if they are the same, then it would add that to the list of addresses to trunk. I'd assume you'd be seeing the same behavior with the new code. Thus, I'm puzzled. That aside, if you don't want the new code to trunk between your DSs on the same server, they should return different owner.

I'm assuming device ids are different for the DSs on different ports?

Quoted 14 lines · Tigran

Hi Olga,
you did not have the answer, however you gave me an important hint! I believe, all our DSes on a single host generate the same server owner during exchange-id. I guess, this can be the reason, why client decides to talk to an other DS.
Tigran.
----- Original Message -----
Quoted 3 lines
From: "Mkrtchyan, Tigran" <[email protected]>
To: "Olga Kornievskaia" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickson=

" <[email protected]>

Quoted 9 lines

Quoted 9 lines
Sent: Monday, March 20, 2017 9:51:21 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Hi Olga,
----- Original Message -----
Quoted 3 lines
From: "Olga Kornievskaia" <[email protected]>
To: "Mkrtchyan, Tigran" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickso=

Quoted 22 lines

Quoted 22 lines
Quoted 4 lines
<[email protected]>
Sent: Monday, March 20, 2017 9:14:34 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Quoted 8 lines
Hi Tigran,
While I don't have an answer to your question, I'd like to point out that in 4.9 is when Andy's session trunking patches when in.
I'm curious this client that's now talking to the DS at port 24006 instead of 24005, did it before also earlier correctly (legally) talked to DS that was on 24006?
Yes, earlier during testing it had legal access to DS on port 24006.
Tigran.
Quoted 5 lines
Quoted 2 lines · Tigran

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Dear (p)NFS-ors,
we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:
If client uses different DSes on the same host, then at some point it =

starts

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to send data to the wrong port number:
Client <=3D> MDS:
    1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call O=

PEN DH:

Quoted 2 lines

    0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
    2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =

(Call In 1) OPEN

Quoted 2 lines

    StateID: 0xec18
    3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call S=

ETATTR FH: 0x6ccf3dfa

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
    4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =

(Call In 3) SETATTR

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
    5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call L=

AYOUTGET

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
    6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =

(Call In 5) LAYOUTGET

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
    7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call G=

ETDEVINFO

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
    8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =

(Call In 7) GETDEVINFO

Quoted 22 lines

        Opcode: GETDEVINFO (47)
            Status: NFS4_OK (0)
            layout type: LAYOUT4_NFSV4_1_FILES (1)
            device index: 0
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            notification bitmap: 6
            notification bitmap: 0
    [Main Opcode: GETDEVINFO (47)]

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
    9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call T=

EST_STATEID

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
   10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =

(Call In 9)

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
TEST_STATEID
NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
client <=3D> DS
$ tshark -r ds-write.pcap  -n -z conv,tcp
    1   0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WR=

ITE StateID: 0xff01

Quoted 2 lines

    Offset: 0 Len: 3968
    2   0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (=

Call In 1) WRITE

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Status: NFS4ERR_BAD_STATEID
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Quoted 3 lines

TCP Conversations
Filter:<No Filter>

                                                           |       <- =

     | |       ->      | |     Total     |    Relative    |   Duration   |
>>>>                                                            | Frames  B=
ytes | | Frames  Bytes | | Frames  Bytes |      Start     |
>>>>                                                            | |
>>>> 131.169.51.50:24006        <-> 131.169.251.53:847               1     =

4240

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
1 168 2 4408 0.000000000 0.0001
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
NOTICE, that it talks to DS on port 24006!
Is there know fix which is missing in CentOS7? I can't reproduce it wi=

Quoted 15 lines

Quoted 15 lines
Quoted 11 lines
Quoted 7 lines
4.9 kernel (or it's harder to reproduce).
The packages are attached.
Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Mkrtchyan, Tigran

author

2017-03-22 20:27 UTC #6 lore raw

Hi Olga,

----- Original Message -----

Quoted 3 lines

From: "Olga Kornievskaia" <[email protected]>
To: "Mkrtchyan, Tigran" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickson"=

<[email protected]>

Quoted 12 lines

Sent: Wednesday, March 22, 2017 5:04:21 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Hi Tigran,
=20
I still don't have the answer to your question but I'm just puzzled why it "works" with 4.9 (session trunking). New code would check the server owner and if they are the same, then it would add that to the list of addresses to trunk. I'd assume you'd be seeing the same behavior with the new code. Thus, I'm puzzled. That aside, if you don't want the new code to trunk between your DSs on the same server, they should return different owner.

I have no idea why 4.9 works differently. May be this no enough 'load' to trigger trunking. Anyway, I have updated our code to generate different server owners. We still need to test it, before all 600 DSes get updated.

Quoted 2 lines

=20
I'm assuming device ids are different for the DSs on different ports?

Yes, each DS gets a unique id.

Thanks,
Tigran.

Quoted 17 lines

=20
Quoted 14 lines · Tigran
Hi Olga,
you did not have the answer, however you gave me an important hint! I believe, all our DSes on a single host generate the same server owner during exchange-id. I guess, this can be the reason, why client decides to talk to an other DS.
Tigran.
----- Original Message -----
Quoted 3 lines
From: "Mkrtchyan, Tigran" <[email protected]>
To: "Olga Kornievskaia" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dickso=

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
<[email protected]>
Sent: Monday, March 20, 2017 9:51:21 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Hi Olga,
----- Original Message -----
Quoted 3 lines
From: "Olga Kornievskaia" <[email protected]>
To: "Mkrtchyan, Tigran" <[email protected]>
Cc: "Linux NFS Mailing list" <[email protected]>, "Steve Dicks=

on"

Quoted 22 lines

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
<[email protected]>
Sent: Monday, March 20, 2017 9:14:34 PM
Subject: Re: pNFS: invalid IP:port selection when talks to DS
Quoted 8 lines
Hi Tigran,
While I don't have an answer to your question, I'd like to point out that in 4.9 is when Andy's session trunking patches when in.
I'm curious this client that's now talking to the DS at port 24006 instead of 24005, did it before also earlier correctly (legally) talked to DS that was on 24006?
Yes, earlier during testing it had legal access to DS on port 24006.
Tigran.
Quoted 5 lines
Quoted 2 lines · Tigran

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Dear (p)NFS-ors,
we observe VERY unpleasant situation with pNFS in the production. Our hosts run multiple DSes on different ports, usually 24001-24009. With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes a wrong port number when talks to data server:
If client uses different DSes on the same host, then at some point it=

starts

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to send data to the wrong port number:
Client <=3D> MDS:
    1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =

OPEN DH:

Quoted 2 lines

    0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
    2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=

(Call In 1) OPEN

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
    StateID: 0xec18
    3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =

SETATTR FH: 0x6ccf3dfa

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
    4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=

(Call In 3) SETATTR

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
    5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =

LAYOUTGET

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
    6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=

(Call In 5) LAYOUTGET

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
    7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =

GETDEVINFO

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
    8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=

(Call In 7) GETDEVINFO

Quoted 22 lines

        Opcode: GETDEVINFO (47)
            Status: NFS4_OK (0)
            layout type: LAYOUT4_NFSV4_1_FILES (1)
            device index: 0
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            r_netid: tcp
                length: 3
                contents: tcp
                fill bytes: opaque data
            r_addr: 131.169.51.50.93.197
                length: 20
                contents: 131.169.51.50.93.197
            notification bitmap: 6
            notification bitmap: 0
    [Main Opcode: GETDEVINFO (47)]

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
    9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =

TEST_STATEID

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
   10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=

(Call In 9)

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
TEST_STATEID
NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
client <=3D> DS
$ tshark -r ds-write.pcap  -n -z conv,tcp
    1   0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call W=

RITE StateID: 0xff01

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
    Offset: 0 Len: 3968
    2   0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply =

(Call In 1) WRITE

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Status: NFS4ERR_BAD_STATEID
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
TCP Conversations
Filter:<No Filter>
                                                           |       <-=

      | |       ->      | |     Total     |    Relative    |   Duration   |
>>>>>                                                            | Frames  =
Bytes | | Frames  Bytes | | Frames  Bytes |      Start     |
>>>>>                                                            | |
>>>>> 131.169.51.50:24006        <-> 131.169.251.53:847               1    =

4240

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
1 168 2 4408 0.000000000 0.0001
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
NOTICE, that it talks to DS on port 24006!
Is there know fix which is missing in CentOS7? I can't reproduce it w=

ith

Quoted 9 lines

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
4.9 kernel (or it's harder to reproduce).
The packages are attached.
Tigran.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" i=

Quoted 10 lines

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html