NFSv4 callback: too many open connections, consider increasing the number of threads.

[resending - with more current address for trond]

Hi Trond et al.

we have a customer who reports getting the message:

        [10879512.275305] NFSv4 callback: too many open  connections,
 consider increasing the number of threads. [10879512.275315] NFSv4 callback:
 too many open  connections, consider increasing the number of threads.
 [10879517.276133] __ratelimit: 133011 callbacks suppressed

Of course you cannot increase the number of threads handling callbacks.

The patch below will get rid of the message - which is presumably mostly just noise - or possibly change it to
NFSv4 callback: too many open connections, consider increasing the max
number of connections if the number of connections gets too high.

This isn't much good as this patch doesn't allow the max number to be increased.
Do we want to make the max number of callback connections configurable, or should we find a way to silence that message.
(For lockd that message is valid as there is a sysctl to change the max number of connections for lockd).

Thanks,
NeilBrown

index 23ff18f..1129c42 100644
@@ -265,6 +265,10 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt

*xprt) ret = -ENOMEM;
goto out_err;
}
+ /* As there is only one thread so we need to over-ride the
+ * default maximum of 80 connections
+ */
+ serv->sv_maxconn = 1024;

ret = svc_bind(serv, net);
if (ret < 0) {

T24gVGh1LCAyMDEyLTA3LTE5IGF0IDE2OjQ2ICsxMDAwLCBOZWlsQnJvd24gd3JvdGU6DQo+IFty ZXNlbmRpbmcgLSB3aXRoIG1vcmUgY3VycmVudCBhZGRyZXNzIGZvciB0cm9uZF0NCj4gDQo+IEhp IFRyb25kIGV0IGFsLg0KPiANCj4gIHdlIGhhdmUgYSBjdXN0b21lciB3aG8gcmVwb3J0cyBnZXR0 aW5nIHRoZSBtZXNzYWdlOg0KPiAgICAgICAgIFsxMDg3OTUxMi4yNzUzMDVdIE5GU3Y0IGNhbGxi YWNrOiB0b28gbWFueSBvcGVuICBjb25uZWN0aW9ucywNCj4gIGNvbnNpZGVyIGluY3JlYXNpbmcg dGhlIG51bWJlciBvZiB0aHJlYWRzLiBbMTA4Nzk1MTIuMjc1MzE1XSBORlN2NCBjYWxsYmFjazoN Cj4gIHRvbyBtYW55IG9wZW4gIGNvbm5lY3Rpb25zLCBjb25zaWRlciBpbmNyZWFzaW5nIHRoZSBu dW1iZXIgb2YgdGhyZWFkcy4NCj4gIFsxMDg3OTUxNy4yNzYxMzNdIF9fcmF0ZWxpbWl0OiAxMzMw MTEgY2FsbGJhY2tzIHN1cHByZXNzZWQNCj4gDQo+IE9mIGNvdXJzZSB5b3UgY2Fubm90IGluY3Jl YXNlIHRoZSBudW1iZXIgb2YgdGhyZWFkcyBoYW5kbGluZyBjYWxsYmFja3MuDQo+IA0KPiBUaGUg cGF0Y2ggYmVsb3cgd2lsbCBnZXQgcmlkIG9mIHRoZSBtZXNzYWdlIC0gd2hpY2ggaXMgcHJlc3Vt YWJseSBtb3N0bHkganVzdA0KPiBub2lzZSAtIG9yIHBvc3NpYmx5IGNoYW5nZSBpdCB0bw0KPiAg ICBORlN2NCBjYWxsYmFjazogdG9vIG1hbnkgb3BlbiBjb25uZWN0aW9ucywgY29uc2lkZXIgaW5j cmVhc2luZyB0aGUgbWF4DQo+IG51bWJlciBvZiBjb25uZWN0aW9ucyBpZiB0aGUgbnVtYmVyIG9m IGNvbm5lY3Rpb25zIGdldHMgdG9vIGhpZ2guDQo+IA0KPiBUaGlzIGlzbid0IG11Y2ggZ29vZCBh cyB0aGlzIHBhdGNoIGRvZXNuJ3QgYWxsb3cgdGhlIG1heCBudW1iZXIgdG8gYmUNCj4gaW5jcmVh c2VkLg0KPiBEbyB3ZSB3YW50IHRvIG1ha2UgdGhlIG1heCBudW1iZXIgb2YgY2FsbGJhY2sgY29u bmVjdGlvbnMgY29uZmlndXJhYmxlLCBvcg0KPiBzaG91bGQgd2UgZmluZCBhIHdheSB0byBzaWxl bmNlIHRoYXQgbWVzc2FnZS4NCj4gKEZvciBsb2NrZCB0aGF0IG1lc3NhZ2UgaXMgdmFsaWQgYXMg dGhlcmUgaXMgYSBzeXNjdGwgdG8gY2hhbmdlIHRoZSBtYXgNCj4gbnVtYmVyIG9mIGNvbm5lY3Rp b25zIGZvciBsb2NrZCkuDQo+IA0KPiBUaGFua3MsDQo+IE5laWxCcm93bg0KPiANCj4gDQo+IGRp ZmYgLS1naXQgYS9mcy9uZnMvY2FsbGJhY2suYyBiL2ZzL25mcy9jYWxsYmFjay5jDQo+IGluZGV4 IDIzZmYxOGYuLjExMjljNDIgMTAwNjQ0DQo+IC0tLSBhL2ZzL25mcy9jYWxsYmFjay5jDQo+ICsr KyBiL2ZzL25mcy9jYWxsYmFjay5jDQo+IEBAIC0yNjUsNiArMjY1LDEwIEBAIGludCBuZnNfY2Fs bGJhY2tfdXAodTMyIG1pbm9ydmVyc2lvbiwgc3RydWN0IHJwY194cHJ0DQo+ICp4cHJ0KSByZXQg PSAtRU5PTUVNOw0KPiAgCQlnb3RvIG91dF9lcnI7DQo+ICAJfQ0KPiArCS8qIEFzIHRoZXJlIGlz IG9ubHkgb25lIHRocmVhZCBzbyB3ZSBuZWVkIHRvIG92ZXItcmlkZSB0aGUNCj4gKwkgKiBkZWZh dWx0IG1heGltdW0gb2YgODAgY29ubmVjdGlvbnMNCj4gKwkgKi8NCj4gKwlzZXJ2LT5zdl9tYXhj b25uID0gMTAyNDsNCj4gIA0KPiAgCXJldCA9IHN2Y19iaW5kKHNlcnYsIG5ldCk7DQo+ICAJaWYg KHJldCA8IDApIHsNCg0KSGkgTmVpbCwNCg0KQ291bGQgeW91IHBsZWFzZSBhZGQgdGhlIG1pc3Np bmcgcy1vLWI/DQoNCkNoZWVycw0KICBUcm9uZA0K

Here 'tis. Thanks.

NeilBrown

From: NeilBrown <[email protected]>
Date: Tue, 31 Jul 2012 14:33:39 +1000
Subject: [PATCH] nfs: increase number of permitted callback connections.

By default a sunrpc service is limited to (N+3)*20 connections where N is the number of threads. This is 80 when N==1. If this number is exceeded a warning is printed suggesting that the number of threads be increased. However with services which run a single thread, this is impossible.

For such services there is a ->sv_maxconn setting that can be used to forcibly increase the limit, and silence the message. This is used by lockd.

The nfs client uses a sunrpc service to handle callbacks and it too is single-threaded, so to avoid the useless messages, and to allow a reasonable number of concurrent connections, we need to set ->sv_maxconn. 1024 seems like a good number.

index 23ff18f..9cb1215 100644
@@ -265,6 +265,10 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt)
 		ret = -ENOMEM;
 		goto out_err;
 	}
+	/* As there is only one thread we need to over-ride the
+	 * default maximum of 80 connections
+	 */
+	serv->sv_maxconn = 1024;
 
 	ret = svc_bind(serv, net);
 	if (ret < 0) {