rdma connection close/reopen every 5 mins
Subsystems (2)
Hello, I have setup NFS/RDMA on our infiniband network. The OS is CentOS 6.8 with kernel 2.6.32-696.1.1.el6.x86_64 and I am using rdma/nfs-rdma packages from the OS version 6.9_4.1. The connection is successful:
10.10.11.249:/newwing on /newwing type nfs (rw,rdma,port=20049,addr=10.10.11.249)
except in /var/log/messages I am always seeing this exactly every 5 minutes:
kernel: rpcrdma: connection to 10.10.11.249:20049 closed (-103)
kernel: rpcrdma: connection to 10.10.11.249:20049 on mlx4_0, memreg 6
slots 32 ird 16
I suppose it is from some service or cron job.... But I wonder how I could keep the connection open so this isn't happening every 5 minutes?
Thanks
Quoted 12 lines · scar
Hello, I have setup NFS/RDMA on our infiniband network. The OS is CentOS 6.8 with kernel 2.6.32-696.1.1.el6.x86_64 and I am using rdma/nfs-rdma packages from the OS version 6.9_4.1. The connection is successful:
10.10.11.249:/newwing on /newwing type nfs (rw,rdma,port=20049,addr=10.10.11.249)
except in
/var/log/messagesI am always seeing this exactly every 5 minutes:
kernel: rpcrdma: connection to 10.10.11.249:20049 closed (-103)
kernel: rpcrdma: connection to 10.10.11.249:20049 onmlx4_0, memreg 6 slots 32 ird 16I suppose it is from some service or cron job.... But I wonder how I could keep the connection open so this isn't happening every 5 minutes?
If it's every 5 minutes, that's probably the RPC idle timeout that is closing the connection.
So this should be entirely harmless.
--
Chuck Lever
Chuck Lever wrote on 04/26/2017 01:32 PM:
Quoted 4 lines
If it's every 5 minutes, that's probably the RPC idle timeout that is closing the connection.
So this should be entirely harmless.
Yes so far it seems harmless, but it is annoying and fills up the log with spam, plus it's not something I see when using NFS/TCP.
thanks
Quoted 10 lines · scar
Chuck Lever wrote on 04/26/2017 01:32 PM:
Quoted 4 lines
If it's every 5 minutes, that's probably the RPC idle timeout that is closing the connection.
So this should be entirely harmless.
Yes so far it seems harmless, but it is annoying and fills up the log with spam, plus it's not something I see when using NFS/TCP.
That's correct, NFS/TCP does not have the "connected / closed" messages, but the idle timeout behavior is the exactly the same.
I tried to remove those messages a few years ago and was told to go pound sand.
The best you can do is create a mechanism to access a file on those mount points more frequently than the idle timeout. That should prevent the connection from closing.
--
Chuck Lever
T24gV2VkLCAyMDE3LTA0LTI2IGF0IDE1OjUyIC0wNzAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g PiBPbiBBcHIgMjYsIDIwMTcsIGF0IDM6NDcgUE0sIHNjYXIgPHNjYXJAZHJpZ29uLmNvbT4gd3Jv dGU6DQo+ID4gDQo+ID4gQ2h1Y2sgTGV2ZXIgd3JvdGUgb24gMDQvMjYvMjAxNyAwMTozMiBQTToN Cj4gPiA+IElmIGl0J3MgZXZlcnkgNSBtaW51dGVzLCB0aGF0J3MgcHJvYmFibHkgdGhlIFJQQyBp ZGxlIHRpbWVvdXQNCj4gPiA+IHRoYXQgaXMgY2xvc2luZyB0aGUgY29ubmVjdGlvbi4NCj4gPiA+ IA0KPiA+ID4gU28gdGhpcyBzaG91bGQgYmUgZW50aXJlbHkgaGFybWxlc3MuDQo+ID4gDQo+ID4g WWVzIHNvIGZhciBpdCBzZWVtcyBoYXJtbGVzcywgYnV0IGl0IGlzIGFubm95aW5nIGFuZCBmaWxs cyB1cCB0aGUNCj4gPiBsb2cgd2l0aCBzcGFtLCBwbHVzIGl0J3Mgbm90IHNvbWV0aGluZyBJIHNl ZSB3aGVuIHVzaW5nIE5GUy9UQ1AuDQo+IA0KPiBUaGF0J3MgY29ycmVjdCwgTkZTL1RDUCBkb2Vz IG5vdCBoYXZlIHRoZSAiY29ubmVjdGVkIC8gY2xvc2VkIg0KPiBtZXNzYWdlcywgYnV0IHRoZSBp ZGxlIHRpbWVvdXQgYmVoYXZpb3IgaXMgdGhlIGV4YWN0bHkgdGhlIHNhbWUuDQo+IA0KPiBJIHRy aWVkIHRvIHJlbW92ZSB0aG9zZSBtZXNzYWdlcyBhIGZldyB5ZWFycyBhZ28gYW5kIHdhcyB0b2xk DQo+IHRvIGdvIHBvdW5kIHNhbmQuDQo+IA0KIA0KDQpCeSBtZT8gSeKAmWQgYmUgaGFwcHkgdG8g dGFrZSBhIHBhdGNoIHRvIHJlbW92ZSB0aG9zZSBtZXNzYWdlcy4gSSBmaW5kIGl0DQphYnNvbHV0 ZWx5IHJpZGljdWxvdXMgdG8gYmUgbG9nZ2luZyBmdWxseSBleHBlY3RlZCBiZWhhdmlvdXIgbGlr ZSB0aGlzLg0KSXQgc2hvdWxkIGF0IGJlc3QgYmUgYSBkcHJpbnRrKCkgYW5kIEknZCBwcmVmZXIg dG8gcmVwbGFjZSBpdCB3aXRoDQp0cmFjZXBvaW50cyBsaWtlIHdlIGRvIGZvciB0aGUgc29ja2V0 IGNvZGUuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWlu ZXIsIFByaW1hcnlEYXRhDQp0cm9uZC5teWtsZWJ1c3RAcHJpbWFyeWRhdGEuY29tDQo=
Quoted 18 lines · Trond Myklebust
Quoted 14 lines · Chuck Lever
Quoted 10 lines · scar
Chuck Lever wrote on 04/26/2017 01:32 PM:
Quoted 4 lines
If it's every 5 minutes, that's probably the RPC idle timeout that is closing the connection.
So this should be entirely harmless.
Yes so far it seems harmless, but it is annoying and fills up the log with spam, plus it's not something I see when using NFS/TCP.
That's correct, NFS/TCP does not have the "connected / closed" messages, but the idle timeout behavior is the exactly the same.
Quoted 6 lines
Quoted 3 lines
I tried to remove those messages a few years ago and was told to go pound sand.
By me?
No, I filed a bug on bugzilla.kernel.org and I think it was Alan Cox (!) who told me to leave these alone.
Quoted 4 lines
I’d be happy to take a patch to remove those messages. I find it absolutely ridiculous to be logging fully expected behaviour like this. It should at best be a
dprintk()and I'd prefer to replace it with tracepoints like we do for the socket code.
I can send a patch for 4.13 that converts these to dprintks for now. Tracepoints are definitely in plan...