Performance of netns with sysfs
There has been some discussion lately about network namespaces and the interaction with sysfs. With the introduction of Guenter's ipcgroup patches, there was also the question of how feasible it is to create thousands of network namespaces. So, I decided to see if I could even create thousands of veth pairs, and if so move them into thousands of network namespaces. I was pleased to see that the system didn't fall over, but found that the process slowed significantly with higher numbers if sysfs was enabled. I thought it would be prudent to post some numbers.
I first tested creating 1000 and 2500 veth pairs, attaching one side to a bridge with and without sysfs. Next I created 2500 network namespaces, along with 2500 veth pairs. One side of each pair was attached to a bridge and the other was moved into the namespace. The results are:
1000 veth pairs: 8x slower with CONFIG_SYSFS
2500 veth pairs 4.5x slower
2500 netns, veth pairs: 6x slower
The tests were done with hal disabled, attaching every third veth device to a different bridge (to overcome the limit of 1023 taps per bridge).
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
A message that is not in the archive (sent off-list or not yet received): 87ocy8thhj.fsf-FLMGYpZoEPULwtHQx/[email protected]
There are two key elements affecting performance when creating large numbers of interfaces:
- full_name_hash() doesn't do a good job in selecting hash buckets
- sysfs uses a linked list to store directory entries, and strcmp() to compare entries.
The sysfs problem may be solved with Eric's sysfs upcoming changes; if not, there are (at least) two ways to improve sysfs performance:
1) store a name hash with each entry, and compare the hash result before comparing the entire name
2) create a per-directory hash table.
full_name_hash() can be improved by replacing the hash, for example as follows.
Old:
return (prevhash + (c << 4) + (c >> 4)) * 11; New: return (prevhash + c) * 41;
Guenter
Quoted 29 lines · Dan Smith
There has been some discussion lately about network namespaces and the interaction with sysfs. With the introduction of Guenter's ipcgroup patches, there was also the question of how feasible it is to create thousands of network namespaces. So, I decided to see if I could even create thousands of veth pairs, and if so move them into thousands of network namespaces. I was pleased to see that the system didn't fall over, but found that the process slowed significantly with higher numbers if sysfs was enabled. I thought it would be prudent to post some numbers.
I first tested creating 1000 and 2500 veth pairs, attaching one side to a bridge with and without sysfs. Next I created 2500 network namespaces, along with 2500 veth pairs. One side of each pair was attached to a bridge and the other was moved into the namespace. The results are:
1000 veth pairs: 8x slower with
CONFIG_SYSFS
2500 veth pairs 4.5x slower
2500 netns, veth pairs: 6x slowerThe tests were done with hal disabled, attaching every third veth device to a different bridge (to overcome the limit of 1023 taps per bridge).
--
Dan Smith
IBM Linux Technology Center
email: danms-r/[email protected]
GR> The sysfs problem may be solved with Eric's sysfs upcoming GR> changes;
Yep, I'll re-run these with Eric's set when it's available.
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
Quoted 8 lines · Guenter Roeck
There are two key elements affecting performance when creating large numbers of interfaces:
-full_name_hash()doesn't do a good job in selecting hash buckets
- sysfs uses a linked list to store directory entries, andstrcmp()to compare entries.The sysfs problem may be solved with Eric's sysfs upcoming changes; if not, there are (at least) two ways to improve sysfs performance:
I have not addressed the slowdown problem. But it is good to hear that it exists and that people care. I thought sysfs was going to slow things down.
My immediate goal is to reduce the locking complexity to that found within /proc.
Quoted 10 lines
1) store a name hash with each entry, and compare the hash result before comparing the entire name
2) create a per-directory hash table.
full_name_hash()can be improved by replacing the hash, for example as follows.
Old:return (prevhash + (c << 4) + (c >> 4)) * 11; New: return (prevhash + c) * 41;
At least with the sanity checking turned on. sysctl will also have slowdowns that are more significant than sysfs when creating network devices. So that should be addressed as well.
Eric