Skip to contentSkip to conversation

How to use SMB Direct

DISCUSSION Xiaoli Feng 7 participants 14 messages Dec 16, 2019 – Dec 20, 2019 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in upstream. I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr() completed -113" Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
	CA type: MT4099
	Number of ports: 2
	Firmware version: 2.42.5000
	Hardware version: 1
	Node GUID: 0xf4521403007be0e0
	System image GUID: 0xf4521403007be0e3
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 29
		LMC: 0
		SM lid: 1
		Capability mask: 0x0259486a
		Port GUID: 0xf4521403007be0e1
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 44
		LMC: 1
		SM lid: 36
		Capability mask: 0x02594868
		Port GUID: 0xf4521403007be0e2
		Link layer: InfiniBand
Signature · 8 lines
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom
#2 lore raw
Quoted 16 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in upstream. I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 5 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr() completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
	CA type: MT4099
	Number of ports: 2
	Firmware version: 2.42.5000
	Hardware version: 1
	Node GUID: 0xf4521403007be0e0
	System image GUID: 0xf4521403007be0e3
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 29
		LMC: 0
		SM lid: 1
		Capability mask: 0x0259486a
		Port GUID: 0xf4521403007be0e1
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 44
		LMC: 1
		SM lid: 36
		Capability mask: 0x02594868
		Port GUID: 0xf4521403007be0e2
		Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#3 lore raw

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:		Link UP, Interface UP	172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 81 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 18 lines

-----Original Message-----
From: [email protected] <[email protected]> On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 6 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr() completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
	CA type: MT4099
	Number of ports: 2
	Firmware version: 2.42.5000
	Hardware version: 1
	Node GUID: 0xf4521403007be0e0
	System image GUID: 0xf4521403007be0e3
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 29
		LMC: 0
		SM lid: 1
		Capability mask: 0x0259486a
		Port GUID: 0xf4521403007be0e1
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 44
		LMC: 1
		SM lid: 36
		Capability mask: 0x02594868
		Port GUID: 0xf4521403007be0e2
		Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#4 lore raw

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 101 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 81 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 18 lines

-----Original Message-----
From: [email protected] <[email protected]> On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 6 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr() completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#5 lore raw

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 131 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 106 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 84 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 20 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 7 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#6 lore raw
Quoted 3 lines · Xiaoli Feng

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

Yeah, that should definitely work. I think Steve, or a colleague, has run through the same set of the tests in our buildbot (manually?) in that configuration.

If you can please test the work-in-progress cifsd kernel server. It should supposedly work with soft-rdma and if it does then that could be a way where we could
integrate SMBDirect testing in the buildbot.

I haven't had any time to research cifsd or set it up and test unfortunately :-(

regards
ronnie sahlberg

Quoted 134 lines

----- Original Message -----

Quoted 131 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 106 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 84 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 20 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 7 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#7 lore raw
In reply to Xiaoli Feng

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 137 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 131 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 106 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 84 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 20 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 7 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#8 lore raw
Quoted 9 lines · Hyeoncheol Lee

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

Thanks Hyeoncheol,

This is very interesting!

We have a buildbot upstream for the cifs client where we test every set of patches before they go to linus :
http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/302

I havent had time to test with your cifsd yet but as you say you can use soft-RoCE
you are saying it will work doing soft-RoCE between one VM running cifs.ko and a basic virtio NIC
to another VM running your cifsd and also using a basic virtio NIC?

I would really want to set up so we can do RDMA tests for cifs.ko, and by accident cifsd, in our buildbot.

regards
ronnie sahlberg

Quoted 147 lines

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 137 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 131 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 106 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 84 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 20 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 7 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#9 lore raw

Hello Ronnie,

2019년 12월 19일 (목) 오후 3:57, ronnie sahlberg <[email protected]>님이 작성:

Quoted 20 lines · Hyeoncheol Lee
Quoted 9 lines

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

Thanks Hyeoncheol,

This is very interesting!

We have a buildbot upstream for the cifs client where we test every set of patches before they go to linus :
http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/302

Quoted 5 lines

I havent had time to test with your cifsd yet but as you say you can use soft-RoCE
you are saying it will work doing soft-RoCE between one VM running cifs.ko and a basic virtio NIC
to another VM running your cifsd and also using a basic virtio NIC?

Yes, I had tested SMB Direct between the two VMs that are configured as you said.

Quoted 4 lines

I would really want to set up so we can do RDMA tests for cifs.ko, and by accident cifsd, in our buildbot.

We will help you. but before that, our cifsd's xfstests test-cases are a different
with your cifs' test-cases. your some test-cases may be failed for cifsd. our test-cases were suggested by Steve in the past.

So we want to know the list of your test-cases and check that the cases will succeed for cifsd.
And while we check the test-cases, the common test-cases between cifs and cifsd can be tested in the buildbot.

Thank you.

Quoted 152 lines

regards
ronnie sahlberg

Quoted 147 lines

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 137 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 131 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 106 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 84 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 20 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 7 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#10 lore raw
In reply to Hyeoncheol Lee

Please read README file to build cifsd

And SMB direct disabled by default.
Enable smb direct support on config.

<M> CIFS server support
[*] Support for SMB Direct protocol

Quoted 172 lines

-----Original Message-----
From: Hyeoncheol Lee <[email protected]>
Sent: Thursday, December 19, 2019 3:48 PM
To: Xiaoli Feng <[email protected]>
Cc: linux-cifs <[email protected]>; Tom Talpey <[email protected]>; [email protected]; ronnie sahlberg <[email protected]>
Subject: Re: [Linux-cifsd-devel] How to use SMB Direct

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://protect2.fireeye.com/url?k=02be8777-5f6a3b1f-02bf0c38-
0cc47a3356b2-b295524a9bc31dd5&u=https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 3 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file server(has rdma hardware) with SMB Direct in

linux client(has rdma hardware)?

Quoted 82 lines

----- Original Message -----

Quoted 80 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 58 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect
failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect

failed
port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory Refer to the
mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 35 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>,
[email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 19 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works in upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 8 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+ kernel.
mount //$RDMA/cifs cifs -o
user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address

issue.

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines

Tom.

Quoted 38 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y $ ibstat CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Red Hat Software (Beijing) Co.,Ltd filesystem-qe Team IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

_______________________________________________
Linux-cifsd-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-cifsd-devel

#11 lore raw
In reply to Hyeoncheol Lee

Hello Hyeoncheol,

When use the cifsd you provide, are there any required for Samba server? Now I have two machines that have Mellanox ConnectX-3. And install 5.5.0-rc1+. If one machine install the cifsd. Then I can use the SMB Direct without the concern of Samba server. Is it right?

As I know, seems there are two projects for SMB Direct(kernel part). One is the cifsd. The others is Metze's repo. Which one is planning to go into linux tree or both are?

https://github.com/cifsd-team/cifsd.git
https://git.samba.org/?p=metze/linux/smbdirect.git;a=shortlog;h=refs/heads/smbdirect-work-in-progress

Thanks.

----- Original Message -----

Quoted 171 lines

From: "Hyeoncheol Lee" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "ronnie sahlberg" <[email protected]>, "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>, [email protected]
Sent: Thursday, December 19, 2019 2:47:40 PM
Subject: Re: How to use SMB Direct

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 144 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file
server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 136 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 110 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 88 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 22 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works
in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The
smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 8 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+
kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

#12 lore raw

The work of Metze's on RDMA/smbdirect extensions to Samba, rely in
part on kernel code. I can help merge that into the kernel but I
would like some agreement between Metze and Long Li (and Tom etc.) on what would be useful to be shared between cifs.ko and the proposed
RDMA helpe driver for Samba. From discussions this fall, it is
probably too early to consider the merge request, but I am very open to following up discussions on this.

The alternative RDMA/smbdirect implementation in cifsd (the proposed smb3 kernel server written by Namjae and other) is very exciting and an interesting topic, but likely unrelated to the Samba server
implementation that Metze is working on. The issues involved in
merging cifsd (the SMB3 kernel server) into the mainline kernel are quite different than those related to Samba's RDMA implementation, and instead mostly have to deal with addressing review comments from others (I have sent namjae various suggestions which he appears to be
addressing). More review feedback of the cifsd kernel server will be
very helpful - but the main issues are not RDMA related.

Quoted 190 lines · Xiaoli Feng

Hello Hyeoncheol,

When use the cifsd you provide, are there any required for Samba server? Now I have two machines that have Mellanox ConnectX-3. And install 5.5.0-rc1+. If one machine install the cifsd. Then I can use the SMB Direct without the concern of Samba server. Is it right?

As I know, seems there are two projects for SMB Direct(kernel part). One is the cifsd. The others is Metze's repo. Which one is planning to go into linux tree or both are?

https://github.com/cifsd-team/cifsd.git
https://git.samba.org/?p=metze/linux/smbdirect.git;a=shortlog;h=refs/heads/smbdirect-work-in-progress

Thanks.

----- Original Message -----

Quoted 171 lines

From: "Hyeoncheol Lee" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "ronnie sahlberg" <[email protected]>, "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>, [email protected]
Sent: Thursday, December 19, 2019 2:47:40 PM
Subject: Re: How to use SMB Direct

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 144 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file
server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 136 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 110 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 88 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 22 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works
in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The
smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 8 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+
kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom

Signature · 3 lines
Thanks,

Steve
#13 lore raw

Am 20.12.19 um 04:26 schrieb Steve French:

Quoted 7 lines

The work of Metze's on RDMA/smbdirect extensions to Samba, rely in
part on kernel code. I can help merge that into the kernel but I
would like some agreement between Metze and Long Li (and Tom etc.) on what would be useful to be shared between cifs.ko and the proposed
RDMA helpe driver for Samba. From discussions this fall, it is
probably too early to consider the merge request, but I am very open to following up discussions on this.

Yes, it's too early. Sadly I still don't have funding to drive this further. But once I have time for it again I'll try to get this to a state where we no longer have memory leaks and pass the microsoft testsuite and manual testing. Once that's done it should be pretty easy to plug this also into cifs.ko (I already have patches and it worked basically a few month ago) and also into the cifsd code. So that we can compare how the implementations perform and if mine is able to perform in a similar way we can start to think about merging it to the kernel have just one implementation for the SMB-Direct protocol that is usable for all 4 cases (cifs.ko and smbclient as client and Samba's smbd and the kernel cifsd as server).

metze

#14 lore raw
In reply to Xiaoli Feng

Hello Xiaoli,

2019년 12월 20일 (금) 오후 12:00, Xiaoli Feng <[email protected]>님이 작성:

Quoted 8 lines

Hello Hyeoncheol,

When use the cifsd you provide, are there any required for Samba server? Now I have two machines that have Mellanox ConnectX-3. And install 5.5.0-rc1+. If one machine install the cifsd. Then I can use the SMB Direct without the concern of Samba server. Is it right?

Yes, you can use the SMB Direct without Samba server.

Following are instructions for cifsd setup.
And cifsd utils and a detailed description can be found in
https://github.com/cifsd-team/cifsd-tools

- Build and install cifsd kernel driver
modprobe cifsd

- Create user/password for SMB share cifsuseradd -a <Enter USERNAME for SMB share access> Enter password for SMB share access

- Create /etc/cifs/smb.conf file, add SMB share in smb.conf file Refer smb.conf.example

- Start cifsd user space daemon
cifsd

- Access share from Windows or Linux using CIFS

Thank you.

Quoted 182 lines

As I know, seems there are two projects for SMB Direct(kernel part). One is the cifsd. The others is Metze's repo. Which one is planning to go into linux tree or both are?

https://github.com/cifsd-team/cifsd.git
https://git.samba.org/?p=metze/linux/smbdirect.git;a=shortlog;h=refs/heads/smbdirect-work-in-progress

Thanks.

----- Original Message -----

Quoted 171 lines

From: "Hyeoncheol Lee" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "ronnie sahlberg" <[email protected]>, "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>, [email protected]
Sent: Thursday, December 19, 2019 2:47:40 PM
Subject: Re: How to use SMB Direct

Hello Xiaoli,

cifsd is an in-kernel SMB server which supports SMB Direct. If you want to test SMB Direct with cifs, you can use cifsd.

Currently we have tested SMB Direct between two old fashioned Mellanox ConnectX devices which are connected directly, and between two soft RoCE devices in kernel.

You can get cifsd's code from the following url, and If you have any questions, we will help you.

https://github.com/cifsd-team/cifsd.git

Thank you.

2019년 12월 19일 (목) 오전 11:35, Xiaoli Feng <[email protected]>님이 작성:

Quoted 144 lines

Thanks Ronnie for the information. Last question: Is it supported that mount the windows file
server(has rdma hardware) with SMB Direct in linux client(has rdma hardware)?

----- Original Message -----

Quoted 136 lines

From: "ronnie sahlberg" <[email protected]>
To: "Xiaoli Feng" <[email protected]>
Cc: "Tom Talpey" <[email protected]>, "linux-cifs" <[email protected]>
Sent: Wednesday, December 18, 2019 4:12:32 PM
Subject: Re: How to use SMB Direct

I don't think samba supports SMB Direct.
Metze used to have a private repo with some experimental patches but I don't think it
has landed in official samba yet.

For a linux server, there is an experimental kernel based server called cifsd which should support SMBDirect
but is incomplete in other areas. It is not part of linus tree yet (but soon I hope).

Windows servers support SMBDirect but I am not aware of any soft-rdma support so you might be
limited to just using real hw for any tests.

Linux serverside SMBD support is not ready afaik.

Quoted 110 lines · Xiaoli Feng

Hello Tom,

Then I try to use IP to mount, it show this error:

[79912.177783] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed
port=5445
[79912.220723] CIFS VFS: _smbd_get_connection:1740 rdma_connect failed

port=445

Client:

$ mount //172.31.0.250/cifs cifs -o user=root,password=redhat,rdma
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Server:

$ ib addr |grep 172.31.0.250/
mlx4_ib0:               Link UP, Interface UP   172.31.0.250/24

Samba version is 4.11.2. And firewalld is stopped.

----- Original Message -----

Quoted 88 lines

From: "Tom Talpey" <[email protected]>
To: "Xiaoli Feng" <[email protected]>, [email protected]
Sent: Tuesday, December 17, 2019 1:11:24 AM
Subject: RE: How to use SMB Direct

Quoted 22 lines

-----Original Message-----
From: [email protected]
<[email protected]>
On
Behalf Of Xiaoli Feng
Sent: Monday, December 16, 2019 10:30 AM
To: [email protected]
Subject: [EXTERNAL] How to use SMB Direct

Hello guys,

I'd like to test SMB Direct. But it's failed. I'm not sure if it works
in
upstream.
I setup samba server on one rdma machine with 5.5.0-rc1+ kernel. The
smb.conf is:
[cifs]
path=/mnt/cifs
writeable=yes

Quoted 8 lines

Then I try to mount the share on another rdma machine with 5.5.0-rc1+
kernel.
mount //$RDMA/cifs cifs -o user=root,password=$password,rdma

It's failed because of "CIFS VFS: smbd_create_id:614 rdma_resolve_addr()
completed -113"

Errno 113 is "no route to host". Sounds like a network or address issue.

Tom.

Quoted 45 lines

Does SMB Direct work fine in upstream?

Thanks.

$ cat /boot/config-5.5.0-rc1+ |grep SMB_DIRECT
CONFIG_CIFS_SMB_DIRECT=y
$ ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.42.5000
    Hardware version: 1
    Node GUID: 0xf4521403007be0e0
    System image GUID: 0xf4521403007be0e3
    Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 56
            Base lid: 29
            LMC: 0
            SM lid: 1
            Capability mask: 0x0259486a
            Port GUID: 0xf4521403007be0e1
            Link layer: InfiniBand
    Port 2:
            State: Active
            Physical state: LinkUp
            Rate: 40
            Base lid: 44
            LMC: 1
            SM lid: 36
            Capability mask: 0x02594868
            Port GUID: 0xf4521403007be0e2
            Link layer: InfiniBand

--
Best regards!
XiaoLi Feng 冯小丽

Red Hat Software (Beijing) Co.,Ltd
filesystem-qe Team
IRC:xifeng,#channel: fs-qe
Tel:+86-10-8388112
9/F, Raycom